[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:16:59 dist-test-slave-dmw7 krb5kdc[1548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:16:59 dist-test-slave-dmw7 krb5kdc[1548](info): set up 2 sockets
Sep 23 22:16:59 dist-test-slave-dmw7 krb5kdc[1548](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 22:17:00.251717 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:00 dist-test-slave-dmw7 krb5kdc[1548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:00.377626 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:33597
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:00.457391 1561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:00.457651 1561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:00.457692 1561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:00.458877 1561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:00.458946 1561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:00.458971 1561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:00.458989 1561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:00.459021 1561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:00.460480 1561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:33597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:00.460764 1561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:00.461027 1561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:00.466526 1561 server_base.cc:1034] running on GCE node
W20260923 22:17:00.467861 1566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:00.468380 1567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:00.468550 1569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:00.468875 1561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:00.469177 1561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:00.470340 1561 hybrid_clock.cc:648] HybridClock initialized: now 1790201820470331 us; error 38 us; skew 500 ppm
Sep 23 22:17:00 dist-test-slave-dmw7 krb5kdc[1548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:00.472918 1561 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:00.473451 1561 webserver.cc:460] Webserver started at http://127.1.129.126:35735/ using document root <none> and password file <none>
I20260923 22:17:00.473699 1561 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:00.473769 1561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:00.473917 1561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:00.475095 1561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "6ccee4466ea643acb37087361d7c205b"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "e24b8b97fd4457ccfa9a384865f251ff"
server_key_iv: "0d5699caaeab35eb14bf550faf6b4d5c"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.475488 1561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "6ccee4466ea643acb37087361d7c205b"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "e24b8b97fd4457ccfa9a384865f251ff"
server_key_iv: "0d5699caaeab35eb14bf550faf6b4d5c"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.476897 1561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:00.477707 1576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.477882 1561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:00.478005 1561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "6ccee4466ea643acb37087361d7c205b"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "e24b8b97fd4457ccfa9a384865f251ff"
server_key_iv: "0d5699caaeab35eb14bf550faf6b4d5c"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.478080 1561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:00.493669 1561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:00.495733 1561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:00.495911 1561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:00.500078 1561 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:33597
I20260923 22:17:00.500113 1628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:33597 every 8 connection(s)
I20260923 22:17:00.500509 1561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:00.501449 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:00.502745 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 1561
I20260923 22:17:00.502852 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:00.503484 1541 external_mini_cluster.cc:1428] Setting key c861a1bdd76e7de6d0b012624fd87bd5
I20260923 22:17:00.503979 1629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b: Bootstrap starting.
I20260923 22:17:00.504658 1629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:00.504997 1629 log.cc:824] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:00.505827 1629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b: No bootstrap required, opened a new log
I20260923 22:17:00.507936 1629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccee4466ea643acb37087361d7c205b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33597 } }
I20260923 22:17:00.508116 1629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:00.508153 1629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccee4466ea643acb37087361d7c205b, State: Initialized, Role: FOLLOWER
I20260923 22:17:00.508285 1629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccee4466ea643acb37087361d7c205b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33597 } }
I20260923 22:17:00.508368 1629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:00.508404 1629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:00.508442 1629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:00.509600 1629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccee4466ea643acb37087361d7c205b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33597 } }
I20260923 22:17:00.509712 1629 leader_election.cc:302] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ccee4466ea643acb37087361d7c205b; no voters:
I20260923 22:17:00.509902 1629 leader_election.cc:288] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:00.510146 1634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:00.510159 1629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:00.510304 1634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [term 1 LEADER]: Becoming Leader. State: Replica: 6ccee4466ea643acb37087361d7c205b, State: Running, Role: LEADER
I20260923 22:17:00.510406 1634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccee4466ea643acb37087361d7c205b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33597 } }
I20260923 22:17:00.510826 1634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ccee4466ea643acb37087361d7c205b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccee4466ea643acb37087361d7c205b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33597 } } }
I20260923 22:17:00.510908 1634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:00.511031 1634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ccee4466ea643acb37087361d7c205b. Latest consensus state: current_term: 1 leader_uuid: "6ccee4466ea643acb37087361d7c205b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccee4466ea643acb37087361d7c205b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33597 } } }
I20260923 22:17:00.511076 1634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:00.511191 1643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:00.511693 1643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:00.513267 1643 catalog_manager.cc:1347] Generated new cluster ID: 56cae4bb072c4cfbbeec136fb150cbd4
I20260923 22:17:00.513324 1643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:00 dist-test-slave-dmw7 krb5kdc[1548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:00.523043 1632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:00.504443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.1.129.126:33597)
0923 22:17:00.504786 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:00.504795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:00.504906 (+ 111us) server_negotiation.cc:407] Connection header received
0923 22:17:00.507909 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:00.507916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:00.507966 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:00.508095 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:00.510371 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.511888 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.513404 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.513978 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.514077 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:00.518357 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:00.518361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:00.518362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:00.518373 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:00.520968 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:00.521455 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:00.521458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:00.521459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:00.521586 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:00.521859 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:00.521860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:00.521861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:00.522182 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:00.522349 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:00.522617 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 22:17:00.522804 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":82,"threads_started":1}
I20260923 22:17:00.531594 1643 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:00.532128 1643 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:00.540925 1643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ccee4466ea643acb37087361d7c205b: Generated new TSK 0
I20260923 22:17:00.541127 1643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:00.567461 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:33597
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:00.650709 1655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:00.650995 1655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:00.651038 1655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:00.652311 1655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:00.652395 1655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:00.652442 1655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:00.654008 1655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:33597
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:00.654320 1655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:00.654579 1655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:00.659627 1663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:00.659638 1661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:00.659657 1660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:00.659757 1655 server_base.cc:1034] running on GCE node
I20260923 22:17:00.660024 1655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:00.660251 1655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:00.661446 1655 hybrid_clock.cc:648] HybridClock initialized: now 1790201820661423 us; error 31 us; skew 500 ppm
Sep 23 22:17:00 dist-test-slave-dmw7 krb5kdc[1548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:00.664211 1655 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:00.664726 1655 webserver.cc:460] Webserver started at http://127.1.129.65:39373/ using document root <none> and password file <none>
I20260923 22:17:00.664958 1655 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:00.665004 1655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:00.665143 1655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:00.666188 1655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "4b72696678374e7f9c066a62bdc8982b"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "c1511c29dbcce5d8a084761d5bcd7283"
server_key_iv: "9781f8a37600f6330fffdefca1f97951"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.666556 1655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "4b72696678374e7f9c066a62bdc8982b"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "c1511c29dbcce5d8a084761d5bcd7283"
server_key_iv: "9781f8a37600f6330fffdefca1f97951"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.667905 1655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:00.668699 1670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.668890 1655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:00.668980 1655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "4b72696678374e7f9c066a62bdc8982b"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "c1511c29dbcce5d8a084761d5bcd7283"
server_key_iv: "9781f8a37600f6330fffdefca1f97951"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.669054 1655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:00.678853 1655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:00.680866 1655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:00.681021 1655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:00.681252 1655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:00.681540 1655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:00.681573 1655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.681595 1655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:00.681630 1655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.687639 1655 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:42469
I20260923 22:17:00.687726 1783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:42469 every 8 connection(s)
I20260923 22:17:00.688035 1655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:00.692553 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 1655
I20260923 22:17:00.692655 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:00.692807 1541 external_mini_cluster.cc:1428] Setting key eb7b3603f1e6cff28aae5c3771e758a9
Sep 23 22:17:00 dist-test-slave-dmw7 krb5kdc[1548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:00.698302 1632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:00.688854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:48057 (local address 127.1.129.126:33597)
0923 22:17:00.688922 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:00.688924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:00.689130 (+ 206us) server_negotiation.cc:407] Connection header received
0923 22:17:00.689635 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:00.689636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:00.689651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:00.689714 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:00.690710 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.691245 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.692230 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.692515 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.692598 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:00.695358 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:00.695360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:00.695360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:00.695368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:00.696829 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:00.697334 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:00.697335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:00.697335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:00.697386 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:00.697663 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:00.697663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:00.697664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:00.697764 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:00.697813 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:00.698153 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 22:17:00.698239 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:17:00.698614 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:00.688828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:33597 (local address 127.1.129.65:48057)
0923 22:17:00.689002 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:00.689045 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:00.689507 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:00.689801 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:00.689804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:00.690386 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:00.690572 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:00.690576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.691332 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.691333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:00.692133 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:00.692135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.692611 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.692638 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:00.692793 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:00.693327 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:00.693337 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:00.695266 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:00.696912 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:00.696919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:00.696922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:00.697249 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:00.697452 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:00.697453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:00.697453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:00.697596 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:00.697829 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:00.697836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:00.698008 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 22:17:00.698136 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 22:17:00.698322 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260923 22:17:00.699322 1784 heartbeater.cc:347] Connected to a master server at 127.1.129.126:33597
I20260923 22:17:00.699425 1784 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:00.699621 1784 heartbeater.cc:511] Master 127.1.129.126:33597 requested a full tablet report, sending...
I20260923 22:17:00.700126 1593 ts_manager.cc:194] Registered new tserver with Master: 4b72696678374e7f9c066a62bdc8982b (127.1.129.65:42469)
I20260923 22:17:00.700748 1593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:48057
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:00.718394 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:33597
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:00.801218 1789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:00.801468 1789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:00.801505 1789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:00.802774 1789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:00.802840 1789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:00.802886 1789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:00.804363 1789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:33597
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:00.804641 1789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:00.804872 1789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:00.810305 1795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:00.810523 1789 server_base.cc:1034] running on GCE node
W20260923 22:17:00.810312 1794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:00.810452 1797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:00.810966 1789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:00.811285 1789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:00.812456 1789 hybrid_clock.cc:648] HybridClock initialized: now 1790201820812443 us; error 28 us; skew 500 ppm
Sep 23 22:17:00 dist-test-slave-dmw7 krb5kdc[1548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:00.814896 1789 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:00.815320 1789 webserver.cc:460] Webserver started at http://127.1.129.66:33563/ using document root <none> and password file <none>
I20260923 22:17:00.815563 1789 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:00.815631 1789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:00.815790 1789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:00.816823 1789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "23efae08ba2a42fe826cee2079e4eafc"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "70a1e33403956267892fbd55ad8b0db2"
server_key_iv: "e84a5e1a8faf11caa8397b84e22b3eab"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.817183 1789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "23efae08ba2a42fe826cee2079e4eafc"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "70a1e33403956267892fbd55ad8b0db2"
server_key_iv: "e84a5e1a8faf11caa8397b84e22b3eab"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.818485 1789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:00.819231 1804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.819383 1789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.819483 1789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "23efae08ba2a42fe826cee2079e4eafc"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "70a1e33403956267892fbd55ad8b0db2"
server_key_iv: "e84a5e1a8faf11caa8397b84e22b3eab"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.819567 1789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:00.837272 1789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:00.840121 1789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:00.840312 1789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:00.840575 1789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:00.840898 1789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:00.840953 1789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.840998 1789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:00.841028 1789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.847824 1789 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:43105
I20260923 22:17:00.847879 1917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:43105 every 8 connection(s)
I20260923 22:17:00.848219 1789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:00.853436 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 1789
I20260923 22:17:00.853556 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:00.853713 1541 external_mini_cluster.cc:1428] Setting key 5a8bc91e29bf484da305977f87a12798
Sep 23 22:17:00 dist-test-slave-dmw7 krb5kdc[1548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:00.858181 1632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:00.849085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:42369 (local address 127.1.129.126:33597)
0923 22:17:00.849177 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:00.849180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:00.849387 (+ 207us) server_negotiation.cc:407] Connection header received
0923 22:17:00.849876 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:00.849877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:00.849886 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:00.849940 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:00.850893 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.851384 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.852368 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.852551 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.852601 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:00.855008 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:00.855009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:00.855010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:00.855017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:00.856592 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:00.857041 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:00.857042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:00.857043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:00.857103 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:00.857440 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:00.857441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:00.857442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:00.857596 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:00.857665 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:00.857937 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 22:17:00.858036 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 22:17:00.858220 1920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:00.849048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:33597 (local address 127.1.129.66:42369)
0923 22:17:00.849229 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:00.849266 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:00.849776 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:00.849998 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:00.850001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:00.850588 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:00.850819 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:00.850823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.851470 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.851470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:00.852290 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:00.852292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:00.852624 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:00.852624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:00.852735 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:00.853308 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:00.853316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:00.854916 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:00.856698 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:00.856706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:00.856709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:00.856949 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:00.857220 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:00.857221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:00.857221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:00.857357 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:00.857667 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:00.857672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:00.857798 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 22:17:00.857894 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 22:17:00.858031 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260923 22:17:00.858863 1918 heartbeater.cc:347] Connected to a master server at 127.1.129.126:33597
I20260923 22:17:00.858961 1918 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:00.859176 1918 heartbeater.cc:511] Master 127.1.129.126:33597 requested a full tablet report, sending...
I20260923 22:17:00.859649 1593 ts_manager.cc:194] Registered new tserver with Master: 23efae08ba2a42fe826cee2079e4eafc (127.1.129.66:43105)
I20260923 22:17:00.860212 1593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:42369
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:00.881145 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:33597
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:00.967303 1923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:00.967577 1923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:00.967618 1923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:00.968786 1923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:00.968870 1923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:00.968920 1923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:00.970472 1923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:33597
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:00.970798 1923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:00.971045 1923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:00.976521 1928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:00.976513 1929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:00.976513 1931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:00.976740 1923 server_base.cc:1034] running on GCE node
I20260923 22:17:00.977052 1923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:00.977340 1923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:00.978506 1923 hybrid_clock.cc:648] HybridClock initialized: now 1790201820978486 us; error 35 us; skew 500 ppm
Sep 23 22:17:00 dist-test-slave-dmw7 krb5kdc[1548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:00.981220 1923 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:00.981823 1923 webserver.cc:460] Webserver started at http://127.1.129.67:43481/ using document root <none> and password file <none>
I20260923 22:17:00.982079 1923 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:00.982133 1923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:00.982286 1923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:00.983419 1923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "da592bef86634a44bf767cfa9e4e4cbe"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "c67513033330273b04f5b4a633b1fe0b"
server_key_iv: "c5a9a79b8b4ae5baaf7eb1744055ceed"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.983791 1923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "da592bef86634a44bf767cfa9e4e4cbe"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "c67513033330273b04f5b4a633b1fe0b"
server_key_iv: "c5a9a79b8b4ae5baaf7eb1744055ceed"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.985164 1923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:00.985904 1938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:00.986073 1923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:00.986189 1923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "da592bef86634a44bf767cfa9e4e4cbe"
format_stamp: "Formatted at 2026-09-23 22:17:00 on dist-test-slave-dmw7"
server_key: "c67513033330273b04f5b4a633b1fe0b"
server_key_iv: "c5a9a79b8b4ae5baaf7eb1744055ceed"
server_key_version: "encryptionkey@0"
I20260923 22:17:00.986279 1923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:01.013564 1923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:01.015915 1923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:01.016081 1923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:01.016321 1923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:01.016659 1923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:01.016700 1923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:01.016741 1923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:01.016767 1923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:01.023355 1923 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44411
I20260923 22:17:01.023447 2051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44411 every 8 connection(s)
I20260923 22:17:01.023774 1923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:01.026865 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 1923
I20260923 22:17:01.026933 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:01.027108 1541 external_mini_cluster.cc:1428] Setting key ec5f3929191a0d112edf9e8c199bd421
Sep 23 22:17:01 dist-test-slave-dmw7 krb5kdc[1548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:01.033885 1632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:01.024624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:49673 (local address 127.1.129.126:33597)
0923 22:17:01.024690 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:01.024692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:01.024843 (+ 151us) server_negotiation.cc:407] Connection header received
0923 22:17:01.025309 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:01.025310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:01.025320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:01.025383 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:01.026274 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:01.026970 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:01.028857 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:01.029044 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:01.029098 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:01.031219 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:01.031221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:01.031221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:01.031227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:01.032577 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:01.032912 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:01.032913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:01.032913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:01.032961 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:01.033274 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:01.033275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:01.033275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:01.033392 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:01.033464 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:01.033749 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 22:17:01.033822 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:17:01.034031 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:01.024542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:33597 (local address 127.1.129.67:49673)
0923 22:17:01.024697 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:01.024740 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:01.025217 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:01.025436 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:01.025439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:01.025979 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:01.026185 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:01.026190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:01.027094 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:01.027095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:01.028785 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:01.028787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:01.029113 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:01.029114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:01.029231 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:01.029731 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:01.029738 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:01.031139 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:01.032650 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:01.032655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:01.032656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:01.032850 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:01.033039 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:01.033040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:01.033040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:01.033182 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:01.033475 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:01.033481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:01.033609 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 22:17:01.033704 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 22:17:01.033830 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260923 22:17:01.034706 2052 heartbeater.cc:347] Connected to a master server at 127.1.129.126:33597
I20260923 22:17:01.034821 2052 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:01.034991 2052 heartbeater.cc:511] Master 127.1.129.126:33597 requested a full tablet report, sending...
I20260923 22:17:01.035372 1593 ts_manager.cc:194] Registered new tserver with Master: da592bef86634a44bf767cfa9e4e4cbe (127.1.129.67:44411)
I20260923 22:17:01.035754 1593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:49673
I20260923 22:17:01.039376 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 22:17:01 dist-test-slave-dmw7 krb5kdc[1548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:01 dist-test-slave-dmw7 krb5kdc[1548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:01.061937 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:01.054321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.1.129.65:42469)
0923 22:17:01.054500 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:01.054508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:01.054522 (+ 14us) server_negotiation.cc:407] Connection header received
0923 22:17:01.054545 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:01.054547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:01.054560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:01.054660 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:01.055213 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:01.055942 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:01.056686 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:01.057004 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:01.057084 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:01.059239 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:01.059247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:01.059247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:01.059254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:01.060712 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:01.061102 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:01.061105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:01.061105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:01.061172 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:01.061366 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:01.061367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:01.061367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:01.061531 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:01.061705 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:01.061746 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 22:17:01.061822 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
W20260923 22:17:01.062315 1698 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53730
Sep 23 22:17:01 dist-test-slave-dmw7 krb5kdc[1548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:01 dist-test-slave-dmw7 krb5kdc[1548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:01.084764 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:01.077314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.1.129.65:42469)
0923 22:17:01.077394 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:01.077398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:01.077428 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:17:01.077504 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:01.077505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:01.077513 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:01.077566 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:01.078104 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:01.078596 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:01.079385 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:01.079590 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:01.079655 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:01.082125 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:01.082126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:01.082126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:01.082133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:01.083669 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:01.084073 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:01.084074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:01.084074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:01.084133 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:01.084336 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:01.084336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:01.084336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:01.084434 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:01.084526 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:01.084596 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 22:17:01.084661 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 22:17:01.085361 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 1655
I20260923 22:17:01.090140 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 1789
I20260923 22:17:01.096385 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 1923
I20260923 22:17:01.101197 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 1561
2026-09-23T22:17:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1897 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:01 dist-test-slave-dmw7 krb5kdc[2066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:01 dist-test-slave-dmw7 krb5kdc[2066](info): set up 2 sockets
Sep 23 22:17:01 dist-test-slave-dmw7 krb5kdc[2066](info): commencing operation
krb5kdc: starting...
W20260923 22:17:02.146391 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:02.269011 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:45487
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:45487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:02.365577 2079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:02.365852 2079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:02.365897 2079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:02.367827 2079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:02.367898 2079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:02.367924 2079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:02.367947 2079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:02.367980 2079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:02.369661 2079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:45487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:02.369968 2079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:02.370231 2079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:02.375486 2085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:02.375469 2087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:02.375567 2079 server_base.cc:1034] running on GCE node
W20260923 22:17:02.375475 2084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:02.375875 2079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:02.376212 2079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:02.377399 2079 hybrid_clock.cc:648] HybridClock initialized: now 1790201822377379 us; error 30 us; skew 500 ppm
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:02.380041 2079 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:02.380569 2079 webserver.cc:460] Webserver started at http://127.1.129.126:41389/ using document root <none> and password file <none>
I20260923 22:17:02.380805 2079 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:02.380874 2079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:02.381026 2079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:02.382081 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "2395bf903b34476e96745deb3813e0b8"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "3a4d2da5caa475222b303bbc12c017de"
server_key_iv: "009e87c02bb7687c3cc64c893204e2eb"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.382452 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "2395bf903b34476e96745deb3813e0b8"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "3a4d2da5caa475222b303bbc12c017de"
server_key_iv: "009e87c02bb7687c3cc64c893204e2eb"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.383924 2079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:02.384641 2094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.384819 2079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:02.384915 2079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "2395bf903b34476e96745deb3813e0b8"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "3a4d2da5caa475222b303bbc12c017de"
server_key_iv: "009e87c02bb7687c3cc64c893204e2eb"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.385000 2079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:02.398468 2079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:02.400646 2079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:02.400811 2079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:02.404767 2079 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:45487
I20260923 22:17:02.404803 2146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:45487 every 8 connection(s)
I20260923 22:17:02.405153 2079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:02.406021 2147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:02.408285 2147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8: Bootstrap starting.
I20260923 22:17:02.408872 2147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:02.409161 2147 log.cc:824] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:02.409891 2147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8: No bootstrap required, opened a new log
I20260923 22:17:02.411541 2147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2395bf903b34476e96745deb3813e0b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45487 } }
I20260923 22:17:02.411680 2147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:02.411705 2147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2395bf903b34476e96745deb3813e0b8, State: Initialized, Role: FOLLOWER
I20260923 22:17:02.411881 2147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2395bf903b34476e96745deb3813e0b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45487 } }
I20260923 22:17:02.411954 2147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:02.411976 2147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:02.412050 2147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:02.412561 2147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2395bf903b34476e96745deb3813e0b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45487 } }
I20260923 22:17:02.412671 2147 leader_election.cc:302] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2395bf903b34476e96745deb3813e0b8; no voters:
I20260923 22:17:02.412842 2147 leader_election.cc:288] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:02.412876 2150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:02.413021 2150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [term 1 LEADER]: Becoming Leader. State: Replica: 2395bf903b34476e96745deb3813e0b8, State: Running, Role: LEADER
I20260923 22:17:02.413110 2147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:02.413111 2150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2395bf903b34476e96745deb3813e0b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45487 } }
I20260923 22:17:02.413554 2151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2395bf903b34476e96745deb3813e0b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2395bf903b34476e96745deb3813e0b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45487 } } }
I20260923 22:17:02.413640 2151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:02.413682 2152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2395bf903b34476e96745deb3813e0b8. Latest consensus state: current_term: 1 leader_uuid: "2395bf903b34476e96745deb3813e0b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2395bf903b34476e96745deb3813e0b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45487 } } }
I20260923 22:17:02.413746 2152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:02.414000 2157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:02.414358 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 2079
I20260923 22:17:02.414443 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:02.414618 1541 external_mini_cluster.cc:1428] Setting key 1067078fe08e5f08011a119638ea3df4
I20260923 22:17:02.415236 2157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:02.417012 2157 catalog_manager.cc:1347] Generated new cluster ID: a21771e5f63740b99cd3f19e53a82948
I20260923 22:17:02.417061 2157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:02.427156 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.415325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.1.129.126:45487)
0923 22:17:02.415587 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:02.415596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:02.415628 (+ 32us) server_negotiation.cc:407] Connection header received
0923 22:17:02.415884 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:02.415893 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:02.415957 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:02.416103 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:02.416735 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.418120 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.419279 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.419955 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.420066 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.422997 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:02.423001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:02.423002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:02.423014 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:02.425051 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.425457 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.425461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.425461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.425592 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.425814 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.425815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.425816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.426148 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:02.426294 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:02.426767 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 22:17:02.426915 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260923 22:17:02.438146 2157 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:02.438963 2157 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:02.443182 2157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2395bf903b34476e96745deb3813e0b8: Generated new TSK 0
I20260923 22:17:02.443390 2157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:02.470376 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45487
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:02.551546 2173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:02.551791 2173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:02.551829 2173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:02.552982 2173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:02.553049 2173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:02.553095 2173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:02.554566 2173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:02.554850 2173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:02.555083 2173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:02.560225 2173 server_base.cc:1034] running on GCE node
W20260923 22:17:02.560191 2181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:02.560194 2178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:02.560195 2179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:02.560671 2173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:02.560982 2173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:02.562145 2173 hybrid_clock.cc:648] HybridClock initialized: now 1790201822562141 us; error 35 us; skew 500 ppm
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:02.564802 2173 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:02.565366 2173 webserver.cc:460] Webserver started at http://127.1.129.65:42755/ using document root <none> and password file <none>
I20260923 22:17:02.565598 2173 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:02.565665 2173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:02.565809 2173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:02.566941 2173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "dcb062d796a5436fba7d6c7bca9cc494"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "4a4769e6eb17376381113de8de385746"
server_key_iv: "5121e7c66781cd6d4c04170ef3a90b8f"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.567306 2173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "dcb062d796a5436fba7d6c7bca9cc494"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "4a4769e6eb17376381113de8de385746"
server_key_iv: "5121e7c66781cd6d4c04170ef3a90b8f"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.568636 2173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:02.569418 2188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.569604 2173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:02.569700 2173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "dcb062d796a5436fba7d6c7bca9cc494"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "4a4769e6eb17376381113de8de385746"
server_key_iv: "5121e7c66781cd6d4c04170ef3a90b8f"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.569793 2173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:02.588768 2173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:02.591274 2173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:02.591454 2173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:02.591693 2173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:02.592008 2173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:02.592061 2173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.592095 2173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:02.592123 2173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.597769 2173 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39095
I20260923 22:17:02.597849 2301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39095 every 8 connection(s)
I20260923 22:17:02.598170 2173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:02.605695 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 2173
I20260923 22:17:02.605777 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:02.605949 1541 external_mini_cluster.cc:1428] Setting key 606d43ccc13d1d49ab3b17c2f4127d6c
I20260923 22:17:02.607789 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.599004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:39901 (local address 127.1.129.126:45487)
0923 22:17:02.599073 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:02.599075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:02.599260 (+ 185us) server_negotiation.cc:407] Connection header received
0923 22:17:02.600184 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:02.600184 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:02.600196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:02.600258 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:02.600833 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.601317 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.602367 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.602555 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.602605 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.604766 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:02.604767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:02.604768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:02.604774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:02.606372 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.606869 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.606870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.606870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.606920 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.607192 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.607193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.607193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.607291 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:02.607340 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:02.607658 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 22:17:02.607719 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:17:02.608098 2304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.598969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45487 (local address 127.1.129.65:39901)
0923 22:17:02.599127 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:02.599169 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:02.600096 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:02.600316 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:02.600319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:02.600561 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:02.600760 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.600765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.601418 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.601419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.602284 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.602287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.602701 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.602703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.602832 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.603308 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:02.603316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:02.604674 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:02.606438 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:02.606445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:02.606447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:02.606789 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:02.606985 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:02.606986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:02.606986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:02.607126 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:02.607353 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:02.607359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:02.607521 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 22:17:02.607641 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 22:17:02.607819 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260923 22:17:02.608512 2302 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45487
I20260923 22:17:02.608603 2302 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:02.608800 2302 heartbeater.cc:511] Master 127.1.129.126:45487 requested a full tablet report, sending...
I20260923 22:17:02.609278 2111 ts_manager.cc:194] Registered new tserver with Master: dcb062d796a5436fba7d6c7bca9cc494 (127.1.129.65:39095)
I20260923 22:17:02.609900 2111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:39901
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:02.632913 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45487
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:02.712566 2307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:02.712805 2307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:02.712841 2307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:02.714148 2307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:02.714210 2307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:02.714253 2307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:02.715914 2307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:02.716190 2307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:02.716426 2307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:02.721481 2307 server_base.cc:1034] running on GCE node
W20260923 22:17:02.721452 2313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:02.721423 2315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:02.721432 2312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:02.721925 2307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:02.722189 2307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:02.723352 2307 hybrid_clock.cc:648] HybridClock initialized: now 1790201822723340 us; error 25 us; skew 500 ppm
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:02.725872 2307 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:02.726372 2307 webserver.cc:460] Webserver started at http://127.1.129.66:45649/ using document root <none> and password file <none>
I20260923 22:17:02.726604 2307 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:02.726703 2307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:02.726856 2307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:02.727973 2307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "7a8e08296e704e0a900ed107a6f13c21"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "fc199d0f1068fb0c43a95adb3e700297"
server_key_iv: "15d3968180c31dc7d1ebf2c099c5018b"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.728341 2307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "7a8e08296e704e0a900ed107a6f13c21"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "fc199d0f1068fb0c43a95adb3e700297"
server_key_iv: "15d3968180c31dc7d1ebf2c099c5018b"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.729688 2307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:02.730479 2322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.730700 2307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:02.730803 2307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "7a8e08296e704e0a900ed107a6f13c21"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "fc199d0f1068fb0c43a95adb3e700297"
server_key_iv: "15d3968180c31dc7d1ebf2c099c5018b"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.730888 2307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:02.753768 2307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:02.756320 2307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:02.756500 2307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:02.756749 2307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:02.757068 2307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:02.757122 2307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.757164 2307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:02.757200 2307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.762887 2307 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:33463
I20260923 22:17:02.762969 2435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:33463 every 8 connection(s)
I20260923 22:17:02.763340 2307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:02.767977 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 2307
I20260923 22:17:02.768088 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:02.768262 1541 external_mini_cluster.cc:1428] Setting key d633b7253a42d126698370f1145a28bd
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:02.773682 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.764278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:48235 (local address 127.1.129.126:45487)
0923 22:17:02.764350 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:02.764351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:02.764535 (+ 184us) server_negotiation.cc:407] Connection header received
0923 22:17:02.765508 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:02.765509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:02.765519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:02.765599 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:02.766289 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.766815 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.767865 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.768056 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.768201 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.770814 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:02.770816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:02.770816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:02.770823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:02.772155 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.772679 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.772681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.772681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.772744 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.772995 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.772996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.772996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.773119 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:02.773193 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:02.773475 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 22:17:02.773560 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:17:02.773862 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.764244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45487 (local address 127.1.129.66:48235)
0923 22:17:02.764399 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:02.764441 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:02.765368 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:02.765706 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:02.765709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:02.765967 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:02.766170 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.766174 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.766909 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.766910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.767734 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.767736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.768202 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.768203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.768354 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.768937 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:02.768947 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:02.770711 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:02.772296 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:02.772304 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:02.772306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:02.772613 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:02.772811 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:02.772812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:02.772813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:02.772927 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:02.773201 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:02.773207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:02.773348 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 22:17:02.773456 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 22:17:02.773617 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260923 22:17:02.774329 2436 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45487
I20260923 22:17:02.774437 2436 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:02.774714 2436 heartbeater.cc:511] Master 127.1.129.126:45487 requested a full tablet report, sending...
I20260923 22:17:02.775132 2111 ts_manager.cc:194] Registered new tserver with Master: 7a8e08296e704e0a900ed107a6f13c21 (127.1.129.66:33463)
I20260923 22:17:02.775666 2111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:48235
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:02.795390 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45487
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:02.876559 2441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:02.876775 2441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:02.876797 2441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:02.877883 2441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:02.877919 2441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:02.877944 2441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:02.879410 2441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:02.879639 2441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:02.879832 2441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:02.884775 2441 server_base.cc:1034] running on GCE node
W20260923 22:17:02.884735 2446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:02.884871 2449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:02.884732 2447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:02.885250 2441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:02.885504 2441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:02.886685 2441 hybrid_clock.cc:648] HybridClock initialized: now 1790201822886672 us; error 29 us; skew 500 ppm
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:02.892652 2441 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:02.893107 2441 webserver.cc:460] Webserver started at http://127.1.129.67:40107/ using document root <none> and password file <none>
I20260923 22:17:02.893293 2441 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:02.893334 2441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:02.893424 2441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:02.894411 2441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "14b21a50f77d4898afc00ae839cfdc2c"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "efb471c10cb1239ea264f21d86dba545"
server_key_iv: "2092a876da1f1460ad88909bd78410e1"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.894779 2441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "14b21a50f77d4898afc00ae839cfdc2c"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "efb471c10cb1239ea264f21d86dba545"
server_key_iv: "2092a876da1f1460ad88909bd78410e1"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.896018 2441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:02.896641 2456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.896809 2441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:02.896911 2441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "14b21a50f77d4898afc00ae839cfdc2c"
format_stamp: "Formatted at 2026-09-23 22:17:02 on dist-test-slave-dmw7"
server_key: "efb471c10cb1239ea264f21d86dba545"
server_key_iv: "2092a876da1f1460ad88909bd78410e1"
server_key_version: "encryptionkey@0"
I20260923 22:17:02.896984 2441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:02.917824 2441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:02.919989 2441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:02.920161 2441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:02.920401 2441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:02.920714 2441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:02.920770 2441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.920814 2441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:02.920843 2441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:02.926743 2441 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:35701
I20260923 22:17:02.926776 2569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:35701 every 8 connection(s)
I20260923 22:17:02.927177 2441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:02.930366 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 2441
I20260923 22:17:02.930477 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:02.930675 1541 external_mini_cluster.cc:1428] Setting key c59e5beb269b09b4884ed837acf18f6f
Sep 23 22:17:02 dist-test-slave-dmw7 krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:02.936760 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.927965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:59041 (local address 127.1.129.126:45487)
0923 22:17:02.928014 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:02.928016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:02.928244 (+ 228us) server_negotiation.cc:407] Connection header received
0923 22:17:02.929145 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:02.929146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:02.929158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:02.929225 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:02.929822 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.930321 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.931732 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.931923 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.932027 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.934130 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:02.934132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:02.934132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:02.934140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:02.935435 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.935830 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.935831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.935831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.935892 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.936138 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.936138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.936139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.936252 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:02.936326 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:02.936585 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 22:17:02.936672 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:02.936942 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.927962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45487 (local address 127.1.129.67:59041)
0923 22:17:02.928118 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:02.928155 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:02.929051 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:02.929299 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:02.929303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:02.929565 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:02.929748 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.929752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.930439 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.930440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.931642 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.931645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.932032 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.932033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.932166 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.932632 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:02.932641 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:02.934038 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:02.935527 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:02.935533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:02.935535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:02.935755 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:02.935959 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:02.935960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:02.935960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:02.936067 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:02.936317 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:02.936323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:02.936457 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 22:17:02.936563 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 22:17:02.936713 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260923 22:17:02.937410 2570 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45487
I20260923 22:17:02.937506 2570 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:02.937640 2570 heartbeater.cc:511] Master 127.1.129.126:45487 requested a full tablet report, sending...
I20260923 22:17:02.937981 2111 ts_manager.cc:194] Registered new tserver with Master: 14b21a50f77d4898afc00ae839cfdc2c (127.1.129.67:35701)
I20260923 22:17:02.938360 2111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:59041
I20260923 22:17:02.942296 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:02.949029 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.943820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.1.129.126:45487)
0923 22:17:02.943861 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:02.943863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:02.943981 (+ 118us) server_negotiation.cc:407] Connection header received
0923 22:17:02.944015 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:02.944016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:02.944024 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:02.944073 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:02.944556 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.944988 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.945606 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.945777 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.945843 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.946993 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:02.946994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:02.946995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:02.947000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:02.948062 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.948371 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.948372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.948372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.948427 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:02.948584 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:02.948585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:02.948585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:02.948715 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:02.948783 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:02.948874 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 22:17:02.948931 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:17:02.951320 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:02.952829 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:02.959825 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.955644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.1.129.67:35701)
0923 22:17:02.956213 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:02.956218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:02.956228 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:02.956271 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:02.956273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:02.956289 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:02.956360 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:02.956580 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.957714 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.958877 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.959411 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.959497 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.959556 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:02.959682 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 22:17:02.959740 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":75,"threads_started":1}
I20260923 22:17:02.959864 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.955355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39095 (local address 127.0.0.1:48198)
0923 22:17:02.955511 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:02.955552 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:02.955616 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:02.956254 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:02.956258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:02.956260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:02.956337 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.956339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.957401 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.957403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.958594 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.958596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.959559 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.959560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.959623 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.959627 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:02.959657 (+ 30us) client_negotiation.cc:240] Negotiation successful
0923 22:17:02.959755 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":58,"threads_started":1}
I20260923 22:17:02.960047 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.955455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.1.129.65:39095)
0923 22:17:02.955783 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:02.955788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:02.955798 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:02.955839 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:02.955842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:02.955857 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:02.956419 (+ 562us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:02.956533 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.957297 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.958711 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.959456 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.959837 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.959896 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:02.959923 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 22:17:02.959970 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":74,"threads_started":1}
I20260923 22:17:02.960052 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.955433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:33463 (local address 127.0.0.1:52758)
0923 22:17:02.956253 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:02.956262 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:02.956300 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:02.956498 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:02.956499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:02.956501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:02.956539 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.956540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.957794 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.957795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.958937 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.958940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.959847 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.959847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.959905 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.959910 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:02.959956 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 22:17:02.959989 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":27,"thread_start_us":78,"threads_started":1}
I20260923 22:17:02.960106 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.955433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:35701 (local address 127.0.0.1:59632)
0923 22:17:02.955650 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:02.955659 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:02.956172 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:02.956448 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:02.956449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:02.956451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:02.956508 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.956509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.957816 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.957817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.958792 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:02.958795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.959497 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.959497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:02.959573 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.959586 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 22:17:02.959667 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 22:17:02.959789 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":101,"thread_start_us":83,"threads_started":1}
I20260923 22:17:02.960351 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:02.955830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.1.129.66:33463)
0923 22:17:02.955989 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:02.955992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:02.956355 (+ 363us) server_negotiation.cc:407] Connection header received
0923 22:17:02.956399 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:02.956401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:02.956414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:02.956486 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:02.956601 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.957686 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.959016 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:02.959699 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:02.959760 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:02.959812 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:02.960223 (+ 411us) server_negotiation.cc:299] Negotiation successful
0923 22:17:02.960255 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260923 22:17:02.960853 2236 tablet_service.cc:1467] Processing CreateTablet for tablet 0566ca4096204e94b8c80a554a853b82 (DEFAULT_TABLE table=test-table [id=afa048579eb542fb9f28294c986614ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:02.960853 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 0566ca4096204e94b8c80a554a853b82 (DEFAULT_TABLE table=test-table [id=afa048579eb542fb9f28294c986614ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:02.961205 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0566ca4096204e94b8c80a554a853b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:02.961199 2236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0566ca4096204e94b8c80a554a853b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:02.961349 2504 tablet_service.cc:1467] Processing CreateTablet for tablet 0566ca4096204e94b8c80a554a853b82 (DEFAULT_TABLE table=test-table [id=afa048579eb542fb9f28294c986614ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:02.961653 2504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0566ca4096204e94b8c80a554a853b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:02.964267 2587 tablet_bootstrap.cc:492] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c: Bootstrap starting.
I20260923 22:17:02.964649 2589 tablet_bootstrap.cc:492] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21: Bootstrap starting.
I20260923 22:17:02.964980 2587 tablet_bootstrap.cc:654] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:02.965209 2589 tablet_bootstrap.cc:654] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:02.965508 2587 log.cc:824] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:02.966074 2589 log.cc:824] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:02.966306 2587 tablet_bootstrap.cc:492] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c: No bootstrap required, opened a new log
I20260923 22:17:02.966385 2587 ts_tablet_manager.cc:1397] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:02.966409 2588 tablet_bootstrap.cc:492] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494: Bootstrap starting.
I20260923 22:17:02.967060 2588 tablet_bootstrap.cc:654] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:02.967492 2588 log.cc:824] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:02.967947 2589 tablet_bootstrap.cc:492] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21: No bootstrap required, opened a new log
I20260923 22:17:02.968016 2589 ts_tablet_manager.cc:1397] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 22:17:02.968214 2588 tablet_bootstrap.cc:492] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494: No bootstrap required, opened a new log
I20260923 22:17:02.968281 2588 ts_tablet_manager.cc:1397] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:02.968305 2587 raft_consensus.cc:348] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:02.968497 2587 raft_consensus.cc:374] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:02.968540 2587 raft_consensus.cc:729] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b21a50f77d4898afc00ae839cfdc2c, State: Initialized, Role: FOLLOWER
I20260923 22:17:02.968662 2587 consensus_queue.cc:260] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:02.968932 2587 ts_tablet_manager.cc:1428] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:02.969056 2570 heartbeater.cc:503] Master 127.1.129.126:45487 was elected leader, sending a full tablet report...
I20260923 22:17:02.969993 2589 raft_consensus.cc:348] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:02.970163 2589 raft_consensus.cc:374] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:02.970223 2589 raft_consensus.cc:729] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8e08296e704e0a900ed107a6f13c21, State: Initialized, Role: FOLLOWER
I20260923 22:17:02.970202 2588 raft_consensus.cc:348] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:02.970355 2588 raft_consensus.cc:374] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:02.970337 2589 consensus_queue.cc:260] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:02.970397 2588 raft_consensus.cc:729] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb062d796a5436fba7d6c7bca9cc494, State: Initialized, Role: FOLLOWER
I20260923 22:17:02.970505 2588 consensus_queue.cc:260] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:02.970607 2589 ts_tablet_manager.cc:1428] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:02.970664 2436 heartbeater.cc:503] Master 127.1.129.126:45487 was elected leader, sending a full tablet report...
I20260923 22:17:02.970809 2302 heartbeater.cc:503] Master 127.1.129.126:45487 was elected leader, sending a full tablet report...
I20260923 22:17:02.970849 2588 ts_tablet_manager.cc:1428] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260923 22:17:03.013983 2437 tablet.cc:2369] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:03.073199 2594 raft_consensus.cc:482] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:03.073371 2594 raft_consensus.cc:504] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:03.073803 2594 leader_election.cc:288] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcb062d796a5436fba7d6c7bca9cc494 (127.1.129.65:39095), 14b21a50f77d4898afc00ae839cfdc2c (127.1.129.67:35701)
I20260923 22:17:03.076683 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:03.073971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39095 (local address 127.1.129.66:43119)
0923 22:17:03.074025 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:03.074037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:03.074082 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:03.074255 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:03.074256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:03.074258 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:03.074341 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:03.074343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.075028 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.075029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:03.075830 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:03.075832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.076396 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.076397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:03.076483 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:03.076486 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:03.076557 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 22:17:03.076603 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 22:17:03.076683 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:03.074021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:43119 (local address 127.1.129.65:39095)
0923 22:17:03.074089 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:03.074091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:03.074103 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:03.074146 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:03.074147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:03.074164 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:03.074231 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:03.074434 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.074937 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.075933 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.076308 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.076400 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:03.076409 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:03.076562 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 22:17:03.076600 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:03.076800 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:03.074051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:35701 (local address 127.1.129.66:43885)
0923 22:17:03.074240 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:03.074249 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:03.074288 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:03.074459 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:03.074460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:03.074462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:03.074535 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:03.074537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.075225 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.075226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:03.075952 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:03.075954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.076527 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.076528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:03.076600 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:03.076603 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:03.076685 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:17:03.076720 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":119,"threads_started":1}
I20260923 22:17:03.076792 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:03.074280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:43885 (local address 127.1.129.67:35701)
0923 22:17:03.074340 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:03.074341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:03.074348 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:03.074377 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:03.074378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:03.074385 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:03.074452 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:03.074612 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.075106 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.076053 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.076432 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.076514 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:03.076521 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:03.076698 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 22:17:03.076733 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:03.077047 2256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0566ca4096204e94b8c80a554a853b82" candidate_uuid: "7a8e08296e704e0a900ed107a6f13c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb062d796a5436fba7d6c7bca9cc494" is_pre_election: true
I20260923 22:17:03.077162 2524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0566ca4096204e94b8c80a554a853b82" candidate_uuid: "7a8e08296e704e0a900ed107a6f13c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b21a50f77d4898afc00ae839cfdc2c" is_pre_election: true
I20260923 22:17:03.077236 2256 raft_consensus.cc:2454] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8e08296e704e0a900ed107a6f13c21 in term 0.
I20260923 22:17:03.077286 2524 raft_consensus.cc:2454] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8e08296e704e0a900ed107a6f13c21 in term 0.
I20260923 22:17:03.077478 2326 leader_election.cc:302] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8e08296e704e0a900ed107a6f13c21, dcb062d796a5436fba7d6c7bca9cc494; no voters:
I20260923 22:17:03.077617 2594 raft_consensus.cc:2781] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:03.077674 2594 raft_consensus.cc:482] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:03.077713 2594 raft_consensus.cc:3037] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:03.078459 2594 raft_consensus.cc:504] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:03.078619 2594 leader_election.cc:288] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [CANDIDATE]: Term 1 election: Requested vote from peers dcb062d796a5436fba7d6c7bca9cc494 (127.1.129.65:39095), 14b21a50f77d4898afc00ae839cfdc2c (127.1.129.67:35701)
I20260923 22:17:03.078799 2256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0566ca4096204e94b8c80a554a853b82" candidate_uuid: "7a8e08296e704e0a900ed107a6f13c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb062d796a5436fba7d6c7bca9cc494"
I20260923 22:17:03.078881 2256 raft_consensus.cc:3037] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:03.078904 2524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0566ca4096204e94b8c80a554a853b82" candidate_uuid: "7a8e08296e704e0a900ed107a6f13c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b21a50f77d4898afc00ae839cfdc2c"
I20260923 22:17:03.078984 2524 raft_consensus.cc:3037] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:03.079584 2256 raft_consensus.cc:2454] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8e08296e704e0a900ed107a6f13c21 in term 1.
I20260923 22:17:03.079838 2326 leader_election.cc:302] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8e08296e704e0a900ed107a6f13c21, dcb062d796a5436fba7d6c7bca9cc494; no voters:
I20260923 22:17:03.079923 2594 raft_consensus.cc:2781] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:03.080085 2594 raft_consensus.cc:686] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8e08296e704e0a900ed107a6f13c21, State: Running, Role: LEADER
I20260923 22:17:03.080163 2524 raft_consensus.cc:2454] T 0566ca4096204e94b8c80a554a853b82 P 14b21a50f77d4898afc00ae839cfdc2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8e08296e704e0a900ed107a6f13c21 in term 1.
I20260923 22:17:03.080183 2594 consensus_queue.cc:237] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } }
I20260923 22:17:03.081020 2110 catalog_manager.cc:5495] T 0566ca4096204e94b8c80a554a853b82 P 7a8e08296e704e0a900ed107a6f13c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8e08296e704e0a900ed107a6f13c21 (127.1.129.66). New cstate: current_term: 1 leader_uuid: "7a8e08296e704e0a900ed107a6f13c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb062d796a5436fba7d6c7bca9cc494" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8e08296e704e0a900ed107a6f13c21" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14b21a50f77d4898afc00ae839cfdc2c" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35701 } health_report { overall_health: UNKNOWN } } }
W20260923 22:17:03.098861 2303 tablet.cc:2369] T 0566ca4096204e94b8c80a554a853b82 P dcb062d796a5436fba7d6c7bca9cc494: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 22:17:03 dist-test-slave-dmw7 krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:03.108130 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:03.101015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.1.129.65:39095)
0923 22:17:03.101094 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:03.101096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:03.101195 (+ 99us) server_negotiation.cc:407] Connection header received
0923 22:17:03.101267 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:03.101268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:03.101279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:03.101349 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:03.101910 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.102396 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.103208 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.103424 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.103499 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:03.105695 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:03.105704 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:03.105704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:03.105710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:03.107195 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:03.107514 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:03.107517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:03.107517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:03.107569 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:03.107723 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:03.107723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:03.107724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:03.107849 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:03.107949 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:03.108005 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 22:17:03.108065 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 23 22:17:03 dist-test-slave-dmw7 krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:03 dist-test-slave-dmw7 krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:03.130066 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:03.123437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.1.129.65:39095)
0923 22:17:03.123525 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:03.123527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:03.123540 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:17:03.123608 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:03.123609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:03.123619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:03.123679 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:03.124202 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.124707 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.125395 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.125578 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.125629 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:03.127788 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:03.127789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:03.127790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:03.127796 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:03.129144 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:03.129416 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:03.129416 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:03.129417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:03.129465 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:03.129602 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:03.129602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:03.129603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:03.129688 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:03.129764 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:03.129898 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 22:17:03.129958 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 22:17:03 dist-test-slave-dmw7 krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:03 dist-test-slave-dmw7 krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:03.151891 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:03.145370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.1.129.65:39095)
0923 22:17:03.145453 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:03.145455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:03.145554 (+ 99us) server_negotiation.cc:407] Connection header received
0923 22:17:03.145638 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:03.145639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:03.145656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:03.145718 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:03.146254 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.146778 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.147525 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:03.147712 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:03.147791 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:03.149828 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:03.149830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:03.149830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:03.149836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:03.151031 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:03.151321 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:03.151322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:03.151322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:03.151369 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:03.151522 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:03.151522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:03.151522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:03.151603 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:03.151671 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:03.151768 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 22:17:03.151828 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 22:17:03.152348 2216 tablet_service.cc:2908] Rejecting scan request for tablet 0566ca4096204e94b8c80a554a853b82: Uninitialized: clean time has not yet been initialized
I20260923 22:17:03.153126 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 2173
I20260923 22:17:03.158674 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 2307
I20260923 22:17:03.163220 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 2441
I20260923 22:17:03.167533 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 2079
2026-09-23T22:17:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2066 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:03 dist-test-slave-dmw7 krb5kdc[2611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:03 dist-test-slave-dmw7 krb5kdc[2611](info): set up 2 sockets
Sep 23 22:17:03 dist-test-slave-dmw7 krb5kdc[2611](info): commencing operation
krb5kdc: starting...
W20260923 22:17:04.206143 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:04.321774 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:45323
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:45323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:04.402350 2624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:04.402558 2624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:04.402580 2624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:04.403803 2624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:04.403844 2624 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 22:17:04.403858 2624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:04.403872 2624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:04.403883 2624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:04.403939 2624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:04.405392 2624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:45323
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:04.405642 2624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:04.405884 2624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:04.410537 2630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:04.411886 2624 server_base.cc:1034] running on GCE node
W20260923 22:17:04.411958 2629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:04.412227 2632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:04.412498 2624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:04.412809 2624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:04.413976 2624 hybrid_clock.cc:648] HybridClock initialized: now 1790201824413960 us; error 27 us; skew 500 ppm
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:04.416462 2624 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:04.416967 2624 webserver.cc:460] Webserver started at http://127.1.129.126:32789/ using document root <none> and password file <none>
I20260923 22:17:04.417189 2624 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:04.417254 2624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:04.417407 2624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:04.418469 2624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "46b1278e28904fa19704e1e5d7b51641"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "c3c866074ca455cc87bd440af46c405a"
server_key_iv: "cb81e31a30e4c95e9ccb44d1af1c1345"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.418955 2624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "46b1278e28904fa19704e1e5d7b51641"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "c3c866074ca455cc87bd440af46c405a"
server_key_iv: "cb81e31a30e4c95e9ccb44d1af1c1345"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.420378 2624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:04.421092 2639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:04.421254 2624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:04.421355 2624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "46b1278e28904fa19704e1e5d7b51641"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "c3c866074ca455cc87bd440af46c405a"
server_key_iv: "cb81e31a30e4c95e9ccb44d1af1c1345"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.421437 2624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:04.437001 2624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:04.439500 2624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:04.439677 2624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:04.444520 2624 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:45323
I20260923 22:17:04.444564 2701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:45323 every 8 connection(s)
I20260923 22:17:04.444919 2624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:04.445559 2702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:04.446455 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 2624
I20260923 22:17:04.446530 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:04.446749 1541 external_mini_cluster.cc:1428] Setting key e9e24c2d668e7fe6ad976e20de466a70
I20260923 22:17:04.448706 2702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641: Bootstrap starting.
I20260923 22:17:04.449338 2702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:04.449640 2702 log.cc:824] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:04.450385 2702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641: No bootstrap required, opened a new log
I20260923 22:17:04.452548 2702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b1278e28904fa19704e1e5d7b51641" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45323 } }
I20260923 22:17:04.452729 2702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:04.452793 2702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b1278e28904fa19704e1e5d7b51641, State: Initialized, Role: FOLLOWER
I20260923 22:17:04.452970 2702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b1278e28904fa19704e1e5d7b51641" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45323 } }
I20260923 22:17:04.453074 2702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:04.453119 2702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:04.453172 2702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:04.453831 2702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b1278e28904fa19704e1e5d7b51641" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45323 } }
I20260923 22:17:04.453955 2702 leader_election.cc:302] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b1278e28904fa19704e1e5d7b51641; no voters:
I20260923 22:17:04.454162 2702 leader_election.cc:288] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:04.454224 2707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:04.454414 2707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [term 1 LEADER]: Becoming Leader. State: Replica: 46b1278e28904fa19704e1e5d7b51641, State: Running, Role: LEADER
I20260923 22:17:04.454452 2702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:04.454499 2707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b1278e28904fa19704e1e5d7b51641" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45323 } }
I20260923 22:17:04.454946 2708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46b1278e28904fa19704e1e5d7b51641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b1278e28904fa19704e1e5d7b51641" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45323 } } }
I20260923 22:17:04.455026 2708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:04.455906 2709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b1278e28904fa19704e1e5d7b51641. Latest consensus state: current_term: 1 leader_uuid: "46b1278e28904fa19704e1e5d7b51641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b1278e28904fa19704e1e5d7b51641" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45323 } } }
I20260923 22:17:04.456111 2709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641 [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:04.456378 2722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:04.456462 2722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:04.456575 2715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:04.456743 2715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:04.457952 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.447331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.1.129.126:45323)
0923 22:17:04.447520 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.447528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.447555 (+ 27us) server_negotiation.cc:407] Connection header received
0923 22:17:04.447998 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.448004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.448055 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.448209 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:04.448828 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.449873 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.450732 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.451173 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.451299 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.453888 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:04.453892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:04.453893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:04.453905 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:04.455992 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.456431 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.456442 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.456442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.456635 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.456857 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.456858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.456859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.457103 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:04.457244 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:04.457610 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 22:17:04.457747 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260923 22:17:04.458501 2715 catalog_manager.cc:1347] Generated new cluster ID: 2528d0f457554eff85159231f280821b
I20260923 22:17:04.458549 2715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:04.476178 2715 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:04.476725 2715 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:04.484687 2715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641: Generated new TSK 0
I20260923 22:17:04.484911 2715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:04.511066 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45323
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:04.592533 2728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:04.592797 2728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:04.592841 2728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:04.594087 2728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:04.594162 2728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:04.594198 2728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:04.594236 2728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:04.595754 2728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:45323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:04.596053 2728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:04.596297 2728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:04.601248 2734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:04.601358 2728 server_base.cc:1034] running on GCE node
W20260923 22:17:04.601244 2736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:04.601248 2733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:04.601732 2728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:04.601997 2728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:04.603160 2728 hybrid_clock.cc:648] HybridClock initialized: now 1790201824603153 us; error 36 us; skew 500 ppm
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:04.605724 2728 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:04.606237 2728 webserver.cc:460] Webserver started at http://127.1.129.65:46183/ using document root <none> and password file <none>
I20260923 22:17:04.606496 2728 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:04.606565 2728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:04.606731 2728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:04.607877 2728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "889981b69b094ec0a9d670d3e18a3105"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "f39cc886c153fa1a201c2c2cb5ffccef"
server_key_iv: "61c3c2fa62a8aa4e2737f931a6b7e21b"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.608230 2728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "889981b69b094ec0a9d670d3e18a3105"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "f39cc886c153fa1a201c2c2cb5ffccef"
server_key_iv: "61c3c2fa62a8aa4e2737f931a6b7e21b"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.609557 2728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:04.610340 2743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:04.610666 2728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:04.610764 2728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "889981b69b094ec0a9d670d3e18a3105"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "f39cc886c153fa1a201c2c2cb5ffccef"
server_key_iv: "61c3c2fa62a8aa4e2737f931a6b7e21b"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.610852 2728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:04.628082 2728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:04.630074 2728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:04.630236 2728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:04.630817 2728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:04.630879 2728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:04.630913 2728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:04.630939 2728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:04.637770 2728 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39457
I20260923 22:17:04.637859 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39457 every 8 connection(s)
I20260923 22:17:04.638186 2728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:04.641356 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.631318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:37371 (local address 127.1.129.126:45323)
0923 22:17:04.631398 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.631401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.631587 (+ 186us) server_negotiation.cc:407] Connection header received
0923 22:17:04.632060 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.632062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.632074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.632178 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:04.633067 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.633917 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.635546 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.635907 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.636021 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.638897 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:04.638898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:04.638898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:04.638906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:04.640186 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.640566 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.640567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.640567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.640620 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.640839 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.640840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.640840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.640945 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:04.640996 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:04.641216 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 22:17:04.641290 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 22:17:04.641511 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.631301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45323 (local address 127.1.129.65:37371)
0923 22:17:04.631449 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.631488 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.631964 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.632254 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.632258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.632644 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:04.632951 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.632958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.634016 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.634017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.635426 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.635430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.636032 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.636033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.636148 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.636760 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:04.636790 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:04.638808 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:04.640269 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.640276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.640278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.640498 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.640678 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.640679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.640679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.640786 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.640998 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:04.641002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:04.641115 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 22:17:04.641196 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 22:17:04.641321 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260923 22:17:04.642035 2859 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45323
I20260923 22:17:04.642118 2859 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:04.642306 2859 heartbeater.cc:511] Master 127.1.129.126:45323 requested a full tablet report, sending...
I20260923 22:17:04.642867 2655 ts_manager.cc:194] Registered new tserver with Master: 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65:39457)
I20260923 22:17:04.643731 2655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:37371
I20260923 22:17:04.646272 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 2728
I20260923 22:17:04.646374 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:04.646548 1541 external_mini_cluster.cc:1428] Setting key d9b6e2aceb79d0300a3606069fd5e6c5
I20260923 22:17:04.649582 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.643190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45323 (local address 127.1.129.65:33969)
0923 22:17:04.643362 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.643373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.643427 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.643540 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.643540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.643694 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:04.643742 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.643743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.644618 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.644619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.645130 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.645131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.645436 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.645436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.645491 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.645932 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:04.645936 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:04.646290 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:04.648210 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.648211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.648212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.648481 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.648755 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.648756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.648757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.648820 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.649208 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:04.649209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:04.649263 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 22:17:04.649358 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 22:17:04.649474 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":123,"threads_started":1}
I20260923 22:17:04.649600 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.643277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:33969 (local address 127.1.129.126:45323)
0923 22:17:04.643331 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.643333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.643431 (+ 98us) server_negotiation.cc:407] Connection header received
0923 22:17:04.643473 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.643474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.643484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.643537 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:04.643814 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.644493 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.645195 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.645374 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.645427 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.646415 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:04.646416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:04.646416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:04.646422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:04.648124 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.648596 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.648597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.648598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.648659 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.648930 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.648931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.648932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.649117 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:04.649183 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:04.649389 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 22:17:04.649517 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:04.673714 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45323
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:04.757750 2868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:04.758016 2868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:04.758080 2868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:04.759332 2868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:04.759383 2868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:04.759409 2868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:04.759476 2868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:04.760967 2868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:45323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:04.761250 2868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:04.761503 2868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:04.766547 2876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:04.766548 2873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:04.766572 2874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:04.766654 2868 server_base.cc:1034] running on GCE node
I20260923 22:17:04.767011 2868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:04.767272 2868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:04.768414 2868 hybrid_clock.cc:648] HybridClock initialized: now 1790201824768402 us; error 25 us; skew 500 ppm
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:04.771103 2868 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:04.771615 2868 webserver.cc:460] Webserver started at http://127.1.129.66:45929/ using document root <none> and password file <none>
I20260923 22:17:04.771858 2868 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:04.771907 2868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:04.772089 2868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:04.773175 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "41182953edad4ff79696003f0e7c280b"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "c4aade8500df271911f51a63f1e84fef"
server_key_iv: "773fd0b6b8b51f75a2977a93dc6eaea4"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.773558 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "41182953edad4ff79696003f0e7c280b"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "c4aade8500df271911f51a63f1e84fef"
server_key_iv: "773fd0b6b8b51f75a2977a93dc6eaea4"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.775055 2868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:04.775870 2883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:04.776051 2868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:04.776163 2868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "41182953edad4ff79696003f0e7c280b"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "c4aade8500df271911f51a63f1e84fef"
server_key_iv: "773fd0b6b8b51f75a2977a93dc6eaea4"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.776252 2868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:04.786726 2868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:04.789271 2868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:04.789448 2868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:04.789996 2868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:04.790076 2868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:04.790117 2868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:04.790169 2868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:04.797791 2868 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:34299
I20260923 22:17:04.798215 2868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:04.798259 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:34299 every 8 connection(s)
I20260923 22:17:04.798763 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 2868
I20260923 22:17:04.798856 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:04.799032 1541 external_mini_cluster.cc:1428] Setting key ee80f4af2af50d333bdf3049dbc265c5
I20260923 22:17:04.800715 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.790487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:32911 (local address 127.1.129.126:45323)
0923 22:17:04.790563 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.790565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.790856 (+ 291us) server_negotiation.cc:407] Connection header received
0923 22:17:04.791621 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.791622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.791635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.791692 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:04.792627 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.793428 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.794427 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.794695 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.794764 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.797377 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:04.797379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:04.797379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:04.797389 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:04.799270 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.799758 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.799759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.799759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.799812 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.800108 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.800109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.800109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.800207 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:04.800259 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:04.800596 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 22:17:04.800655 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:17:04.801028 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.790469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45323 (local address 127.1.129.66:32911)
0923 22:17:04.790701 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.790756 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.791484 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.791774 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.791779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.792201 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:04.792501 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.792509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.793513 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.793515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.794330 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.794333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.794787 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.794788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.794919 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.795446 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:04.795455 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:04.797279 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:04.799344 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.799351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.799354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.799655 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.799885 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.799886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.799887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.800037 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.800279 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:04.800285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:04.800453 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 22:17:04.800568 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 22:17:04.800756 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260923 22:17:04.801542 2999 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45323
I20260923 22:17:04.801645 2999 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:04.801787 2999 heartbeater.cc:511] Master 127.1.129.126:45323 requested a full tablet report, sending...
I20260923 22:17:04.802532 2656 ts_manager.cc:194] Registered new tserver with Master: 41182953edad4ff79696003f0e7c280b (127.1.129.66:34299)
I20260923 22:17:04.803092 2656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:32911
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
I20260923 22:17:04.809765 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.802735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:48983 (local address 127.1.129.126:45323)
0923 22:17:04.802777 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.802779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.803395 (+ 616us) server_negotiation.cc:407] Connection header received
0923 22:17:04.803431 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.803432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.803444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.803514 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:04.803957 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.804449 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.805606 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.805786 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.805849 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.806913 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:04.806915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:04.806915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:04.806921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:04.808477 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.808799 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.808800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.808801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.808857 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.809028 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.809028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.809029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.809199 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:04.809426 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:04.809589 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 22:17:04.809677 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:04.809780 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.802716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45323 (local address 127.1.129.66:48983)
0923 22:17:04.803313 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.803324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.803389 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.803543 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.803544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.803774 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:04.803865 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.803867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.804560 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.804562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.805503 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.805506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.805871 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.805873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.805957 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.806377 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:04.806383 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:04.806822 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:04.808555 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.808556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.808556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.808709 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.808924 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.808924 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.808925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.808972 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.809413 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:04.809414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:04.809467 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 22:17:04.809562 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 22:17:04.809672 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:04.826095 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45323
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:04.905258 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:04.905508 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:04.905547 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:04.906725 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:04.906797 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:04.906834 3008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:04.906873 3008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:04.908264 3008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:45323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:04.908519 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:04.908739 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:04.914609 3013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:04.914652 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:04.914664 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:04.915089 3008 server_base.cc:1034] running on GCE node
I20260923 22:17:04.915309 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:04.915570 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:04.916735 3008 hybrid_clock.cc:648] HybridClock initialized: now 1790201824916715 us; error 30 us; skew 500 ppm
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:04.919229 3008 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:04.919809 3008 webserver.cc:460] Webserver started at http://127.1.129.67:33901/ using document root <none> and password file <none>
I20260923 22:17:04.920042 3008 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:04.920123 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:04.920243 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:04.921319 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "3c3f7ce07cf7411f95d935839021f4b0"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "a4f55b8bbfb3acea81f705d4b2148ada"
server_key_iv: "68b80b607f6ba70aa027cbaaca4e822f"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.921620 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "3c3f7ce07cf7411f95d935839021f4b0"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "a4f55b8bbfb3acea81f705d4b2148ada"
server_key_iv: "68b80b607f6ba70aa027cbaaca4e822f"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.922935 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:04.923676 3023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:04.923878 3008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:04.923949 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "3c3f7ce07cf7411f95d935839021f4b0"
format_stamp: "Formatted at 2026-09-23 22:17:04 on dist-test-slave-dmw7"
server_key: "a4f55b8bbfb3acea81f705d4b2148ada"
server_key_iv: "68b80b607f6ba70aa027cbaaca4e822f"
server_key_version: "encryptionkey@0"
I20260923 22:17:04.924041 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:04.939781 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:04.941984 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:04.942107 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:04.942757 3008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:04.942804 3008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:04.942828 3008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:04.942842 3008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:04.950405 3008 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44629
I20260923 22:17:04.950529 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44629 every 8 connection(s)
I20260923 22:17:04.950834 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:04.953542 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.943144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:47097 (local address 127.1.129.126:45323)
0923 22:17:04.943213 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.943216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.943507 (+ 291us) server_negotiation.cc:407] Connection header received
0923 22:17:04.944247 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.944266 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.944279 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.944342 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:04.945314 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.945817 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.947573 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.947783 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.947871 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.950620 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:04.950622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:04.950623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:04.950632 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:04.952266 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.952662 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.952663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.952663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.952723 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.952954 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.952955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.952955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.953063 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:04.953150 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:04.953408 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 22:17:04.953479 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:04.953612 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.943144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45323 (local address 127.1.129.67:47097)
0923 22:17:04.943347 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.943402 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.944140 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.944441 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.944445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.944877 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:04.945198 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.945207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.945927 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.945928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.947454 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.947458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.947891 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.947892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.948061 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.948593 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:04.948605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:04.950460 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:04.952362 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.952368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.952370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.952585 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.952802 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.952802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.952803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.952897 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.953141 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:04.953145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:04.953254 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 22:17:04.953341 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:17:04.953460 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260923 22:17:04.954123 3139 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45323
I20260923 22:17:04.954206 3139 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:04.954450 3139 heartbeater.cc:511] Master 127.1.129.126:45323 requested a full tablet report, sending...
I20260923 22:17:04.954913 2656 ts_manager.cc:194] Registered new tserver with Master: 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67:44629)
I20260923 22:17:04.955497 2656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:47097
I20260923 22:17:04.960546 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.955346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:51469 (local address 127.1.129.126:45323)
0923 22:17:04.955388 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.955390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.955446 (+ 56us) server_negotiation.cc:407] Connection header received
0923 22:17:04.955578 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.955579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.955588 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.955642 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:04.956111 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.956674 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.957336 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.957506 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.957554 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.958439 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:04.958440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:04.958440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:04.958445 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:04.959711 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.959999 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.960000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.960000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.960044 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.960195 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.960196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.960196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.960281 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:04.960340 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:04.960439 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 22:17:04.960489 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:04.960559 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.955272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45323 (local address 127.1.129.67:51469)
0923 22:17:04.955393 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.955403 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.955511 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.955649 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.955650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.955860 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:04.956050 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.956052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.956765 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.956766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.957249 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.957251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.957567 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.957567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:04.957636 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.958084 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:04.958089 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:04.958369 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:04.959784 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.959785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.959785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.959937 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.960098 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:04.960099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:04.960099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:04.960146 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:04.960343 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:04.960343 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:04.960376 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 22:17:04.960424 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:17:04.960494 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260923 22:17:04.960793 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 3008
I20260923 22:17:04.960855 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:04.960973 1541 external_mini_cluster.cc:1428] Setting key 8edf71a1959986c0abdd2ffe983ea0f0
I20260923 22:17:04.961799 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:04 dist-test-slave-dmw7 krb5kdc[2611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:04.992962 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.984945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.1.129.126:45323)
0923 22:17:04.984990 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.984992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.985001 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:04.985032 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.985033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.985041 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.985116 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:04.985944 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.986802 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.988073 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:04.988252 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:04.988308 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:04.990578 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:04.990580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:04.990580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:04.990586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:04.992080 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.992434 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.992436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.992436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.992484 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:04.992607 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:04.992608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:04.992608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:04.992694 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:04.992756 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:04.992846 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 22:17:04.992900 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:04.994562 2656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:04.996232 2656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:05.002969 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.998516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44629 (local address 127.0.0.1:38682)
0923 22:17:04.998833 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.998842 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.998895 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.999307 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.999307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.999310 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:04.999351 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.999353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.000594 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.000595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.001693 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.001695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.002601 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.002602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.002727 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.002743 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.002809 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.002874 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260923 22:17:05.002990 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.998956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.1.129.66:34299)
0923 22:17:04.999136 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.999141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.999152 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:04.999192 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.999194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.999211 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.999285 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:04.999494 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.000203 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.001679 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.002476 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.002551 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.002616 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.002847 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.002891 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260923 22:17:05.003199 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.998725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.1.129.67:44629)
0923 22:17:04.999134 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.999137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.999146 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:04.999182 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.999184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.999198 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.999424 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:04.999476 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.000521 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.001764 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.002475 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.002938 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.003006 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.003033 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.003077 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":101,"threads_started":1}
I20260923 22:17:05.003412 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.998768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34299 (local address 127.0.0.1:35940)
0923 22:17:04.998964 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.998974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.999023 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.999348 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.999349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.999351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:04.999414 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.999416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.000308 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.000309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.001578 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.001581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.002668 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.002669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.002730 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.002749 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.002820 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.002874 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":21,"thread_start_us":58,"threads_started":1}
I20260923 22:17:05.003651 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.998516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39457 (local address 127.0.0.1:34448)
0923 22:17:04.998697 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:04.998742 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:04.998808 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:04.999190 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:04.999193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:04.999195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:04.999264 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:04.999266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.001772 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.001773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.002432 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.002433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.003429 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.003430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.003493 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.003498 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.003548 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.003586 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"spinlock_wait_cycles":384,"thread_start_us":87,"threads_started":1}
I20260923 22:17:05.003980 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:04.998632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.1.129.65:39457)
0923 22:17:04.998956 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:04.998961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:04.998973 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:04.999016 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:04.999019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:04.999036 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:04.999112 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:04.999341 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.000506 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.002526 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.003275 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.003343 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.003395 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.003828 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.003875 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":105,"threads_started":1}
I20260923 22:17:05.003893 3073 tablet_service.cc:1467] Processing CreateTablet for tablet 2243549560514ea2b27a62b5b4fa953d (DEFAULT_TABLE table=test-table [id=a5b5d025208f4a10967934a8f55c40cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:05.004269 3073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2243549560514ea2b27a62b5b4fa953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.004199 2933 tablet_service.cc:1467] Processing CreateTablet for tablet 2243549560514ea2b27a62b5b4fa953d (DEFAULT_TABLE table=test-table [id=a5b5d025208f4a10967934a8f55c40cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:05.004545 2933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2243549560514ea2b27a62b5b4fa953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.004571 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 2243549560514ea2b27a62b5b4fa953d (DEFAULT_TABLE table=test-table [id=a5b5d025208f4a10967934a8f55c40cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:05.004904 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2243549560514ea2b27a62b5b4fa953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.007198 3162 tablet_bootstrap.cc:492] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Bootstrap starting.
I20260923 22:17:05.008179 3162 tablet_bootstrap.cc:654] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.008217 3163 tablet_bootstrap.cc:492] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Bootstrap starting.
I20260923 22:17:05.008488 3164 tablet_bootstrap.cc:492] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Bootstrap starting.
I20260923 22:17:05.008522 3162 log.cc:824] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:05.008826 3163 tablet_bootstrap.cc:654] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.009187 3163 log.cc:824] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:05.009210 3164 tablet_bootstrap.cc:654] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.009352 3162 tablet_bootstrap.cc:492] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: No bootstrap required, opened a new log
I20260923 22:17:05.009424 3162 ts_tablet_manager.cc:1397] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:05.009570 3164 log.cc:824] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:05.010004 3163 tablet_bootstrap.cc:492] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: No bootstrap required, opened a new log
I20260923 22:17:05.010138 3163 ts_tablet_manager.cc:1397] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:05.010309 3164 tablet_bootstrap.cc:492] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: No bootstrap required, opened a new log
I20260923 22:17:05.010386 3164 ts_tablet_manager.cc:1397] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:05.011662 3162 raft_consensus.cc:348] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.011839 3162 raft_consensus.cc:374] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.011878 3162 raft_consensus.cc:729] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c3f7ce07cf7411f95d935839021f4b0, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.011760 3164 raft_consensus.cc:348] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.011925 3164 raft_consensus.cc:374] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.011960 3164 raft_consensus.cc:729] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889981b69b094ec0a9d670d3e18a3105, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.012032 3162 consensus_queue.cc:260] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.012101 3164 consensus_queue.cc:260] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.012202 3163 raft_consensus.cc:348] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.012310 3162 ts_tablet_manager.cc:1428] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:05.012355 3164 ts_tablet_manager.cc:1428] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:05.012360 3163 raft_consensus.cc:374] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.012399 3163 raft_consensus.cc:729] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41182953edad4ff79696003f0e7c280b, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.012430 3139 heartbeater.cc:503] Master 127.1.129.126:45323 was elected leader, sending a full tablet report...
I20260923 22:17:05.012429 2859 heartbeater.cc:503] Master 127.1.129.126:45323 was elected leader, sending a full tablet report...
I20260923 22:17:05.012499 3163 consensus_queue.cc:260] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.012722 3163 ts_tablet_manager.cc:1428] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 22:17:05.012985 2999 heartbeater.cc:503] Master 127.1.129.126:45323 was elected leader, sending a full tablet report...
W20260923 22:17:05.049535 3000 tablet.cc:2369] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:05.139287 2860 tablet.cc:2369] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:05.201555 3140 tablet.cc:2369] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:05.301072 3169 raft_consensus.cc:482] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:05.301292 3169 raft_consensus.cc:504] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.301717 3169 leader_election.cc:288] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67:44629), 41182953edad4ff79696003f0e7c280b (127.1.129.66:34299)
I20260923 22:17:05.303557 3170 raft_consensus.cc:482] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:05.303676 3170 raft_consensus.cc:504] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.304016 3170 leader_election.cc:288] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67:44629), 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65:39457)
I20260923 22:17:05.305361 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.302031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:60865 (local address 127.1.129.66:34299)
0923 22:17:05.302093 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.302095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.302246 (+ 151us) server_negotiation.cc:407] Connection header received
0923 22:17:05.302274 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.302275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.302284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.302362 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:05.302510 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.303092 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.304427 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.304999 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.305041 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.305050 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.305245 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.305296 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:05.305361 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.302123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:58779 (local address 127.1.129.67:44629)
0923 22:17:05.302189 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.302191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.302221 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:17:05.302258 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.302259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.302274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.302360 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:05.302522 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.303091 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.304435 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.304990 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.305033 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.305041 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.305243 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.305280 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:05.305560 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.301971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44629 (local address 127.1.129.65:58779)
0923 22:17:05.302147 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.302156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.302210 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.302368 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.302370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.302375 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:05.302441 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.302443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.303178 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.303179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.304331 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.304333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.305098 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.305099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.305172 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.305175 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.305433 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.305481 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260923 22:17:05.305693 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.301932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34299 (local address 127.1.129.65:60865)
0923 22:17:05.302082 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.302092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.302184 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.302368 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.302369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.302371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:05.302434 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.302436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.303180 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.303181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.304325 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.304328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.305105 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.305105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.305167 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.305171 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.305570 (+ 399us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.305612 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":40,"thread_start_us":72,"threads_started":1}
I20260923 22:17:05.306012 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2243549560514ea2b27a62b5b4fa953d" candidate_uuid: "889981b69b094ec0a9d670d3e18a3105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c3f7ce07cf7411f95d935839021f4b0" is_pre_election: true
I20260923 22:17:05.306162 3093 raft_consensus.cc:2454] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889981b69b094ec0a9d670d3e18a3105 in term 0.
I20260923 22:17:05.306398 2744 leader_election.cc:302] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c3f7ce07cf7411f95d935839021f4b0, 889981b69b094ec0a9d670d3e18a3105; no voters:
I20260923 22:17:05.306406 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2243549560514ea2b27a62b5b4fa953d" candidate_uuid: "889981b69b094ec0a9d670d3e18a3105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41182953edad4ff79696003f0e7c280b" is_pre_election: true
I20260923 22:17:05.306492 2953 raft_consensus.cc:2454] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889981b69b094ec0a9d670d3e18a3105 in term 0.
I20260923 22:17:05.306579 3169 raft_consensus.cc:2781] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:05.306740 3169 raft_consensus.cc:482] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:05.306772 3169 raft_consensus.cc:3037] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.307531 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.304168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44629 (local address 127.1.129.66:45403)
0923 22:17:05.304379 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.304390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.304435 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.304652 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.304653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.304667 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:05.304723 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.304724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.305540 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.305540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.306711 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.306714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.307259 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.307260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.307340 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.307344 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.307395 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.307433 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":52,"threads_started":1}
I20260923 22:17:05.307678 3169 raft_consensus.cc:504] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.307731 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.304305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:45403 (local address 127.1.129.67:44629)
0923 22:17:05.304521 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.304523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.304529 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:17:05.304550 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.304550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.304559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.304629 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:05.304810 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.305455 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.306810 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.307175 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.307261 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.307268 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.307613 (+ 345us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.307652 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
I20260923 22:17:05.307839 3169 leader_election.cc:288] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [CANDIDATE]: Term 1 election: Requested vote from peers 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67:44629), 41182953edad4ff79696003f0e7c280b (127.1.129.66:34299)
I20260923 22:17:05.307902 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2243549560514ea2b27a62b5b4fa953d" candidate_uuid: "41182953edad4ff79696003f0e7c280b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c3f7ce07cf7411f95d935839021f4b0" is_pre_election: true
I20260923 22:17:05.307967 3093 raft_consensus.cc:2454] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41182953edad4ff79696003f0e7c280b in term 0.
I20260923 22:17:05.308012 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2243549560514ea2b27a62b5b4fa953d" candidate_uuid: "889981b69b094ec0a9d670d3e18a3105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41182953edad4ff79696003f0e7c280b"
I20260923 22:17:05.308080 2953 raft_consensus.cc:3037] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.308280 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.304296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:45395 (local address 127.1.129.65:39457)
0923 22:17:05.304434 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.304436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.304727 (+ 291us) server_negotiation.cc:407] Connection header received
0923 22:17:05.304827 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.304828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.304835 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.304861 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:05.305060 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.305720 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.307367 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.307907 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.307966 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.307972 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.308170 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.308215 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260923 22:17:05.308380 3092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2243549560514ea2b27a62b5b4fa953d" candidate_uuid: "889981b69b094ec0a9d670d3e18a3105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c3f7ce07cf7411f95d935839021f4b0"
I20260923 22:17:05.308440 3092 raft_consensus.cc:3037] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.308561 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.304183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39457 (local address 127.1.129.66:45395)
0923 22:17:05.304679 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.304689 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.304748 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.304912 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.304913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.304914 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:05.304983 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.304984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.306152 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.306152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.307292 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.307295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.308004 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.308005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.308081 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.308086 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.308454 (+ 368us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.308492 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":39,"threads_started":1}
I20260923 22:17:05.308820 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2243549560514ea2b27a62b5b4fa953d" candidate_uuid: "41182953edad4ff79696003f0e7c280b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889981b69b094ec0a9d670d3e18a3105" is_pre_election: true
I20260923 22:17:05.308929 2813 raft_consensus.cc:2379] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41182953edad4ff79696003f0e7c280b in current term 1: Already voted for candidate 889981b69b094ec0a9d670d3e18a3105 in this term.
I20260923 22:17:05.309027 2953 raft_consensus.cc:2454] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889981b69b094ec0a9d670d3e18a3105 in term 1.
I20260923 22:17:05.309175 2746 leader_election.cc:302] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41182953edad4ff79696003f0e7c280b, 889981b69b094ec0a9d670d3e18a3105; no voters:
I20260923 22:17:05.309254 3169 raft_consensus.cc:2781] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:05.309368 3169 raft_consensus.cc:686] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 1 LEADER]: Becoming Leader. State: Replica: 889981b69b094ec0a9d670d3e18a3105, State: Running, Role: LEADER
I20260923 22:17:05.309376 3092 raft_consensus.cc:2454] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889981b69b094ec0a9d670d3e18a3105 in term 1.
I20260923 22:17:05.309481 3169 consensus_queue.cc:237] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } }
I20260923 22:17:05.309939 2884 leader_election.cc:302] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c3f7ce07cf7411f95d935839021f4b0, 41182953edad4ff79696003f0e7c280b; no voters: 889981b69b094ec0a9d670d3e18a3105
I20260923 22:17:05.310105 3170 raft_consensus.cc:2741] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260923 22:17:05.310352 2656 catalog_manager.cc:5495] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "889981b69b094ec0a9d670d3e18a3105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:05.355684 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.352878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.1.129.65:39457)
0923 22:17:05.352930 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.352932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.352988 (+ 56us) server_negotiation.cc:407] Connection header received
0923 22:17:05.353070 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.353071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.353089 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.353149 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:05.353290 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.353846 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.354814 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.354995 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.355050 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.355239 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:05.355335 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:05.355396 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.355552 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.355608 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:05.357460 3092 raft_consensus.cc:1261] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Refusing update from remote peer 889981b69b094ec0a9d670d3e18a3105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:05.357477 2953 raft_consensus.cc:1261] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Refusing update from remote peer 889981b69b094ec0a9d670d3e18a3105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:05.357793 3169 consensus_queue.cc:1035] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:05.357978 3176 consensus_queue.cc:1035] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:05.360327 3179 mvcc.cc:205] Tried to move back new op lower bound from 7332666676661608448 to 7332666676468367360. Current Snapshot: MvccSnapshot[applied={T|T < 7332666676661608448}]
I20260923 22:17:05.360404 3180 mvcc.cc:205] Tried to move back new op lower bound from 7332666676661608448 to 7332666676468367360. Current Snapshot: MvccSnapshot[applied={T|T < 7332666676661608448}]
I20260923 22:17:05.362529 3181 mvcc.cc:205] Tried to move back new op lower bound from 7332666676661608448 to 7332666676468367360. Current Snapshot: MvccSnapshot[applied={T|T < 7332666676661608448 or (T in {7332666676661608448})}]
I20260923 22:17:05.364198 2656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:05.364312 2656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:05.365072 2656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641: Sending DeleteTablet for 3 replicas of tablet 2243549560514ea2b27a62b5b4fa953d
I20260923 22:17:05.365417 3073 tablet_service.cc:1514] Processing DeleteTablet for tablet 2243549560514ea2b27a62b5b4fa953d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:05 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:38682
I20260923 22:17:05.365460 2933 tablet_service.cc:1514] Processing DeleteTablet for tablet 2243549560514ea2b27a62b5b4fa953d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:05 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:35940
I20260923 22:17:05.365698 2656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 22:17:05.365793 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet 2243549560514ea2b27a62b5b4fa953d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:05 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:34448
W20260923 22:17:05.365866 2656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:05.366003 3189 tablet_replica.cc:330] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: stopping tablet replica
I20260923 22:17:05.366110 3189 raft_consensus.cc:2229] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:05.366225 3189 raft_consensus.cc:2258] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:05.366552 3191 tablet_replica.cc:330] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: stopping tablet replica
I20260923 22:17:05.366888 3189 ts_tablet_manager.cc:1905] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:05.367017 3073 tablet_service.cc:1467] Processing CreateTablet for tablet 4e629a3d8ee14412a5954d1eed5b1c37 (DEFAULT_TABLE table=test-table [id=8ef74389d0224ece97d9ea9f92dcc0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:05.367139 3073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e629a3d8ee14412a5954d1eed5b1c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.367136 2933 tablet_service.cc:1467] Processing CreateTablet for tablet 4e629a3d8ee14412a5954d1eed5b1c37 (DEFAULT_TABLE table=test-table [id=8ef74389d0224ece97d9ea9f92dcc0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:05.367338 2933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e629a3d8ee14412a5954d1eed5b1c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.368600 3162 tablet_bootstrap.cc:492] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: Bootstrap starting.
I20260923 22:17:05.368636 3163 tablet_bootstrap.cc:492] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: Bootstrap starting.
I20260923 22:17:05.366247 3190 tablet_replica.cc:330] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: stopping tablet replica
I20260923 22:17:05.369060 3162 tablet_bootstrap.cc:654] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.369059 3163 tablet_bootstrap.cc:654] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.369086 3190 raft_consensus.cc:2229] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:05.369242 3190 raft_consensus.cc:2258] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:05.367055 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 4e629a3d8ee14412a5954d1eed5b1c37 (DEFAULT_TABLE table=test-table [id=8ef74389d0224ece97d9ea9f92dcc0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:05.369485 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e629a3d8ee14412a5954d1eed5b1c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.369729 3162 tablet_bootstrap.cc:492] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: No bootstrap required, opened a new log
I20260923 22:17:05.369731 3190 ts_tablet_manager.cc:1905] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:05.369776 3162 ts_tablet_manager.cc:1397] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:05.369935 3162 raft_consensus.cc:348] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.369994 3162 raft_consensus.cc:374] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.370012 3162 raft_consensus.cc:729] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c3f7ce07cf7411f95d935839021f4b0, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.370064 3162 consensus_queue.cc:260] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.370164 3162 ts_tablet_manager.cc:1428] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:05.370889 3164 tablet_bootstrap.cc:492] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: Bootstrap starting.
I20260923 22:17:05.371307 3190 ts_tablet_manager.cc:1918] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:05.371358 3190 log.cc:1196] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/2243549560514ea2b27a62b5b4fa953d
I20260923 22:17:05.371392 3164 tablet_bootstrap.cc:654] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.371554 3190 ts_tablet_manager.cc:1939] T 2243549560514ea2b27a62b5b4fa953d P 41182953edad4ff79696003f0e7c280b: Deleting consensus metadata
I20260923 22:17:05.372061 3164 tablet_bootstrap.cc:492] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: No bootstrap required, opened a new log
I20260923 22:17:05.372068 3163 tablet_bootstrap.cc:492] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: No bootstrap required, opened a new log
I20260923 22:17:05.372113 3163 ts_tablet_manager.cc:1397] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 22:17:05.372117 3164 ts_tablet_manager.cc:1397] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:05.372247 3163 raft_consensus.cc:348] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.372273 3164 raft_consensus.cc:348] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.372342 3163 raft_consensus.cc:374] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.372359 3164 raft_consensus.cc:374] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.372364 3163 raft_consensus.cc:729] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41182953edad4ff79696003f0e7c280b, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.372386 3164 raft_consensus.cc:729] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889981b69b094ec0a9d670d3e18a3105, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.372409 3163 consensus_queue.cc:260] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.372450 3164 consensus_queue.cc:260] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.372500 3163 ts_tablet_manager.cc:1428] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:05.372548 3164 ts_tablet_manager.cc:1428] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:05.373020 2642 catalog_manager.cc:4841] TS 41182953edad4ff79696003f0e7c280b (127.1.129.66:34299): tablet 2243549560514ea2b27a62b5b4fa953d (table test-table [id=a5b5d025208f4a10967934a8f55c40cc]) successfully deleted
I20260923 22:17:05.372885 3191 raft_consensus.cc:2229] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:05.373404 3191 raft_consensus.cc:2258] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:05.373560 3189 ts_tablet_manager.cc:1918] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:05.373636 3189 log.cc:1196] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/2243549560514ea2b27a62b5b4fa953d
I20260923 22:17:05.373879 3191 ts_tablet_manager.cc:1905] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:05.373907 3189 ts_tablet_manager.cc:1939] T 2243549560514ea2b27a62b5b4fa953d P 3c3f7ce07cf7411f95d935839021f4b0: Deleting consensus metadata
I20260923 22:17:05.374341 2640 catalog_manager.cc:4841] TS 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67:44629): tablet 2243549560514ea2b27a62b5b4fa953d (table test-table [id=a5b5d025208f4a10967934a8f55c40cc]) successfully deleted
I20260923 22:17:05.375375 3191 ts_tablet_manager.cc:1918] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:05.375464 3191 log.cc:1196] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/2243549560514ea2b27a62b5b4fa953d
I20260923 22:17:05.375690 3191 ts_tablet_manager.cc:1939] T 2243549560514ea2b27a62b5b4fa953d P 889981b69b094ec0a9d670d3e18a3105: Deleting consensus metadata
I20260923 22:17:05.376082 2642 catalog_manager.cc:4841] TS 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65:39457): tablet 2243549560514ea2b27a62b5b4fa953d (table test-table [id=a5b5d025208f4a10967934a8f55c40cc]) successfully deleted
I20260923 22:17:05.705066 3168 raft_consensus.cc:482] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:05.705166 3168 raft_consensus.cc:504] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.705430 3168 leader_election.cc:288] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65:39457), 41182953edad4ff79696003f0e7c280b (127.1.129.66:34299)
I20260923 22:17:05.708504 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.705805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:50291 (local address 127.1.129.65:39457)
0923 22:17:05.705907 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.705909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.705961 (+ 52us) server_negotiation.cc:407] Connection header received
0923 22:17:05.706027 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.706029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.706039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.706108 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:05.706275 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.706811 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.707809 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.708158 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.708240 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.708247 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.708401 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.708437 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 22:17:05.708501 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.705993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:49623 (local address 127.1.129.66:34299)
0923 22:17:05.706035 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.706037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.706047 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:05.706114 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.706115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.706123 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.706181 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:05.706332 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.706828 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.707801 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.708151 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.708237 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.708246 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.708387 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.708422 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:05.708554 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.705885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34299 (local address 127.1.129.67:49623)
0923 22:17:05.705967 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.705978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.706054 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.706195 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.706196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.706199 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:05.706258 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.706259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.706924 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.706925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.707712 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.707713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.708236 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.708236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.708311 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.708314 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.708380 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.708421 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":28,"thread_start_us":41,"threads_started":1}
I20260923 22:17:05.708556 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.705688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39457 (local address 127.1.129.67:50291)
0923 22:17:05.705892 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.705905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.705971 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.706125 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.706126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.706128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:05.706204 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.706206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.706891 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.706892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.707706 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.707707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.708240 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.708241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.708322 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.708326 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.708395 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.708437 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260923 22:17:05.708846 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e629a3d8ee14412a5954d1eed5b1c37" candidate_uuid: "3c3f7ce07cf7411f95d935839021f4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889981b69b094ec0a9d670d3e18a3105" is_pre_election: true
I20260923 22:17:05.708850 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e629a3d8ee14412a5954d1eed5b1c37" candidate_uuid: "3c3f7ce07cf7411f95d935839021f4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41182953edad4ff79696003f0e7c280b" is_pre_election: true
I20260923 22:17:05.708950 2813 raft_consensus.cc:2454] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c3f7ce07cf7411f95d935839021f4b0 in term 0.
I20260923 22:17:05.708964 2953 raft_consensus.cc:2454] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c3f7ce07cf7411f95d935839021f4b0 in term 0.
I20260923 22:17:05.709144 3026 leader_election.cc:302] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c3f7ce07cf7411f95d935839021f4b0, 889981b69b094ec0a9d670d3e18a3105; no voters:
I20260923 22:17:05.709326 3168 raft_consensus.cc:2781] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:05.709375 3168 raft_consensus.cc:482] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:05.709401 3168 raft_consensus.cc:3037] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.710103 3168 raft_consensus.cc:504] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.710233 3168 leader_election.cc:288] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [CANDIDATE]: Term 1 election: Requested vote from peers 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65:39457), 41182953edad4ff79696003f0e7c280b (127.1.129.66:34299)
I20260923 22:17:05.710441 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e629a3d8ee14412a5954d1eed5b1c37" candidate_uuid: "3c3f7ce07cf7411f95d935839021f4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41182953edad4ff79696003f0e7c280b"
I20260923 22:17:05.710427 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e629a3d8ee14412a5954d1eed5b1c37" candidate_uuid: "3c3f7ce07cf7411f95d935839021f4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889981b69b094ec0a9d670d3e18a3105"
I20260923 22:17:05.710503 2953 raft_consensus.cc:3037] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.710523 2813 raft_consensus.cc:3037] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.711225 2813 raft_consensus.cc:2454] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c3f7ce07cf7411f95d935839021f4b0 in term 1.
I20260923 22:17:05.711272 2953 raft_consensus.cc:2454] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c3f7ce07cf7411f95d935839021f4b0 in term 1.
I20260923 22:17:05.711438 3026 leader_election.cc:302] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c3f7ce07cf7411f95d935839021f4b0, 889981b69b094ec0a9d670d3e18a3105; no voters:
I20260923 22:17:05.711536 3168 raft_consensus.cc:2781] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:05.711647 3168 raft_consensus.cc:686] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3c3f7ce07cf7411f95d935839021f4b0, State: Running, Role: LEADER
I20260923 22:17:05.711717 3168 consensus_queue.cc:237] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.712378 2656 catalog_manager.cc:5495] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67). New cstate: current_term: 1 leader_uuid: "3c3f7ce07cf7411f95d935839021f4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } health_report { overall_health: HEALTHY } } }
Sep 23 22:17:05 dist-test-slave-dmw7 krb5kdc[2611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:05.746315 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.739439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.1.129.126:45323)
0923 22:17:05.739578 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.739580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.739668 (+ 88us) server_negotiation.cc:407] Connection header received
0923 22:17:05.739741 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.739742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.739753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.739848 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:05.740313 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.740804 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.741528 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.741711 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.741819 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.743953 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:05.743955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:05.743955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:05.743968 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:05.745140 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:05.745511 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:05.745512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:05.745513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:05.745575 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:05.745756 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:05.745757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:05.745758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:05.745873 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:05.745946 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.746106 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.746239 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260923 22:17:05.746315 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.739355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45323 (local address 127.0.0.1:56340)
0923 22:17:05.739586 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.739597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.739687 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.739849 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.739850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.740163 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:05.740205 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.740207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.740884 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.740885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.741451 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.741453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.741820 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.741821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.741870 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.742479 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:05.742483 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:05.743835 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:05.745222 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:05.745229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:05.745232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:05.745426 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:05.745638 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:05.745639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:05.745639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:05.745697 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:05.745946 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:05.745954 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:05.745996 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.746071 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.746187 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":125,"threads_started":1}
I20260923 22:17:05.748237 2656 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:56340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 22:17:05.748481 2656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:05.749954 3073 tablet_service.cc:1467] Processing CreateTablet for tablet 0825aa4ac40c40a6965e0dabe992dc3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ef4dd676c6b4f7297096e99b90fb120]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:05.749956 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 0825aa4ac40c40a6965e0dabe992dc3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ef4dd676c6b4f7297096e99b90fb120]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:05.750121 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0825aa4ac40c40a6965e0dabe992dc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.750121 3073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0825aa4ac40c40a6965e0dabe992dc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.750795 2933 tablet_service.cc:1467] Processing CreateTablet for tablet 0825aa4ac40c40a6965e0dabe992dc3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ef4dd676c6b4f7297096e99b90fb120]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:05.750952 2933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0825aa4ac40c40a6965e0dabe992dc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:05.751646 3164 tablet_bootstrap.cc:492] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: Bootstrap starting.
I20260923 22:17:05.751959 3164 tablet_bootstrap.cc:654] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.752218 3162 tablet_bootstrap.cc:492] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: Bootstrap starting.
I20260923 22:17:05.752364 3163 tablet_bootstrap.cc:492] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: Bootstrap starting.
I20260923 22:17:05.752681 3162 tablet_bootstrap.cc:654] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.752822 3163 tablet_bootstrap.cc:654] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:05.753177 3164 tablet_bootstrap.cc:492] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: No bootstrap required, opened a new log
I20260923 22:17:05.753221 3164 ts_tablet_manager.cc:1397] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:05.753329 3164 raft_consensus.cc:348] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.753379 3164 raft_consensus.cc:374] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.753392 3164 raft_consensus.cc:729] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889981b69b094ec0a9d670d3e18a3105, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.753384 3163 tablet_bootstrap.cc:492] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: No bootstrap required, opened a new log
I20260923 22:17:05.753418 3163 ts_tablet_manager.cc:1397] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:05.753453 3164 consensus_queue.cc:260] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.753563 3164 ts_tablet_manager.cc:1428] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:05.753582 3169 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.753648 3169 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:05.753983 3162 tablet_bootstrap.cc:492] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: No bootstrap required, opened a new log
I20260923 22:17:05.754033 3162 ts_tablet_manager.cc:1397] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:05.754038 3163 raft_consensus.cc:348] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.754122 3163 raft_consensus.cc:374] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.754153 3163 raft_consensus.cc:729] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41182953edad4ff79696003f0e7c280b, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.754204 3163 consensus_queue.cc:260] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.754182 3162 raft_consensus.cc:348] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.754293 3162 raft_consensus.cc:374] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:05.754323 3162 raft_consensus.cc:729] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c3f7ce07cf7411f95d935839021f4b0, State: Initialized, Role: FOLLOWER
I20260923 22:17:05.754324 3163 ts_tablet_manager.cc:1428] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:05.754344 3170 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.754367 3162 consensus_queue.cc:260] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.754416 3170 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:05.754453 3162 ts_tablet_manager.cc:1428] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:05.754565 3168 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.754650 3168 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:05.782663 3170 raft_consensus.cc:482] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:05.782761 3170 raft_consensus.cc:504] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.782968 3170 leader_election.cc:288] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65:39457), 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67:44629)
I20260923 22:17:05.783190 3092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0825aa4ac40c40a6965e0dabe992dc3f" candidate_uuid: "41182953edad4ff79696003f0e7c280b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c3f7ce07cf7411f95d935839021f4b0" is_pre_election: true
I20260923 22:17:05.783208 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0825aa4ac40c40a6965e0dabe992dc3f" candidate_uuid: "41182953edad4ff79696003f0e7c280b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889981b69b094ec0a9d670d3e18a3105" is_pre_election: true
I20260923 22:17:05.783300 3092 raft_consensus.cc:2454] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41182953edad4ff79696003f0e7c280b in term 0.
I20260923 22:17:05.783310 2813 raft_consensus.cc:2454] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41182953edad4ff79696003f0e7c280b in term 0.
I20260923 22:17:05.783509 2886 leader_election.cc:302] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41182953edad4ff79696003f0e7c280b, 889981b69b094ec0a9d670d3e18a3105; no voters:
I20260923 22:17:05.783617 3170 raft_consensus.cc:2781] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:05.783668 3170 raft_consensus.cc:482] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:05.783699 3170 raft_consensus.cc:3037] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.784379 3170 raft_consensus.cc:504] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.784515 3170 leader_election.cc:288] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [CANDIDATE]: Term 1 election: Requested vote from peers 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65:39457), 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67:44629)
I20260923 22:17:05.784663 3092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0825aa4ac40c40a6965e0dabe992dc3f" candidate_uuid: "41182953edad4ff79696003f0e7c280b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c3f7ce07cf7411f95d935839021f4b0"
I20260923 22:17:05.784663 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0825aa4ac40c40a6965e0dabe992dc3f" candidate_uuid: "41182953edad4ff79696003f0e7c280b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889981b69b094ec0a9d670d3e18a3105"
I20260923 22:17:05.784734 3092 raft_consensus.cc:3037] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.784758 2813 raft_consensus.cc:3037] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:05.785354 3092 raft_consensus.cc:2454] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41182953edad4ff79696003f0e7c280b in term 1.
I20260923 22:17:05.785360 2813 raft_consensus.cc:2454] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41182953edad4ff79696003f0e7c280b in term 1.
I20260923 22:17:05.785518 2884 leader_election.cc:302] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c3f7ce07cf7411f95d935839021f4b0, 41182953edad4ff79696003f0e7c280b; no voters:
I20260923 22:17:05.785612 3170 raft_consensus.cc:2781] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:05.785753 3170 raft_consensus.cc:686] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [term 1 LEADER]: Becoming Leader. State: Replica: 41182953edad4ff79696003f0e7c280b, State: Running, Role: LEADER
I20260923 22:17:05.785815 3170 consensus_queue.cc:237] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } }
I20260923 22:17:05.786170 3209 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: TxnStatusTablet state changed. Reason: New leader 41182953edad4ff79696003f0e7c280b. Latest consensus state: current_term: 1 leader_uuid: "41182953edad4ff79696003f0e7c280b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.786273 3209 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:05.786510 3211 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 22:17:05.786656 3211 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 22:17:05.786547 2655 catalog_manager.cc:5495] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b reported cstate change: term changed from 0 to 1, leader changed from <none> to 41182953edad4ff79696003f0e7c280b (127.1.129.66). New cstate: current_term: 1 leader_uuid: "41182953edad4ff79696003f0e7c280b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:05.792176 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.789307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34299 (local address 127.0.0.1:35950)
0923 22:17:05.789383 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.789400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.789435 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.789600 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.789601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.789602 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:05.789648 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.789651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.790542 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.790543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.791208 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.791211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.791681 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.791682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.791745 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.791760 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:05.792037 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:05.792043 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.792081 (+ 38us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.792121 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 22:17:05.792187 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.789371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.1.129.66:34299)
0923 22:17:05.789417 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.789419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.789430 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:05.789496 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.789497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.789515 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.789602 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:05.789690 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.790436 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.791292 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.791589 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.791663 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.791857 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:05.791970 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:05.792049 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.792088 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.792109 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:17:05.793228 3092 raft_consensus.cc:1261] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Refusing update from remote peer 41182953edad4ff79696003f0e7c280b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:05.793255 2813 raft_consensus.cc:1261] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Refusing update from remote peer 41182953edad4ff79696003f0e7c280b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:05.793458 3209 consensus_queue.cc:1035] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [LEADER]: Connected to new peer: Peer: permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:05.793560 3170 consensus_queue.cc:1035] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:05.794118 3168 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: TxnStatusTablet state changed. Reason: New leader 41182953edad4ff79696003f0e7c280b. Latest consensus state: current_term: 1 leader_uuid: "41182953edad4ff79696003f0e7c280b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.794215 3168 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:05.794303 3169 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: TxnStatusTablet state changed. Reason: New leader 41182953edad4ff79696003f0e7c280b. Latest consensus state: current_term: 1 leader_uuid: "41182953edad4ff79696003f0e7c280b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.794384 3169 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:05.794999 3170 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41182953edad4ff79696003f0e7c280b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.795082 3170 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:05.795171 3170 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41182953edad4ff79696003f0e7c280b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.795225 3170 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 41182953edad4ff79696003f0e7c280b: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:05.795647 3169 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41182953edad4ff79696003f0e7c280b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.795658 3168 tablet_replica.cc:439] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41182953edad4ff79696003f0e7c280b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 } } peers { permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 } } peers { permanent_uuid: "3c3f7ce07cf7411f95d935839021f4b0" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44629 } } }
I20260923 22:17:05.795719 3169 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 889981b69b094ec0a9d670d3e18a3105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:05.795735 3168 tablet_replica.cc:442] T 0825aa4ac40c40a6965e0dabe992dc3f P 3c3f7ce07cf7411f95d935839021f4b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:05.799062 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.796689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.1.129.67:44629)
0923 22:17:05.796747 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.796749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.796757 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:05.796799 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.796800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.796820 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.796897 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:05.797027 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.797483 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.798297 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.798506 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.798591 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.798747 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:05.798851 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:05.798932 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.798974 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.798993 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:17:05.807390 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.804081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:50339 (local address 127.1.129.66:34299)
0923 22:17:05.804135 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.804137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.804224 (+ 87us) server_negotiation.cc:407] Connection header received
0923 22:17:05.804282 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.804283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.804291 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.804335 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:05.804467 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.804902 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.806777 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.806957 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.807015 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.807190 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:05.807215 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:05.807253 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.807328 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.807343 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:05.807446 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.803999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34299 (local address 127.1.129.67:50339)
0923 22:17:05.804147 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.804157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.804224 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.804344 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.804345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.804346 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:05.804411 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.804413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.805265 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.805266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.806692 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.806694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.807031 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.807032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.807097 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.807112 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:05.807271 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:05.807273 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.807325 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.807370 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260923 22:17:05.809737 2953 raft_consensus.cc:1261] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Refusing update from remote peer 3c3f7ce07cf7411f95d935839021f4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:05.809757 2813 raft_consensus.cc:1261] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Refusing update from remote peer 3c3f7ce07cf7411f95d935839021f4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:05.809988 3168 consensus_queue.cc:1035] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889981b69b094ec0a9d670d3e18a3105" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:05.810134 3197 consensus_queue.cc:1035] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41182953edad4ff79696003f0e7c280b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:05.818619 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.816048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:58431 (local address 127.1.129.67:44629)
0923 22:17:05.816085 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.816086 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.816228 (+ 142us) server_negotiation.cc:407] Connection header received
0923 22:17:05.816327 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.816328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.816337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.816381 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:05.816505 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.816966 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.817800 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.818102 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.818169 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.818354 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:05.818393 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:05.818444 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.818525 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.818547 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:17:05.818677 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.816006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44629 (local address 127.1.129.66:58431)
0923 22:17:05.816163 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:05.816176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:05.816257 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:05.816382 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:05.816382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:05.816384 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:05.816446 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.816447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.817042 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.817043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.817653 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:05.817654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.818196 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.818197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:05.818246 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.818286 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:05.818457 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:05.818459 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:05.818520 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:17:05.818563 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260923 22:17:05.825487 2656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:05.825596 2656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:05.826267 2656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46b1278e28904fa19704e1e5d7b51641: Sending DeleteTablet for 3 replicas of tablet 4e629a3d8ee14412a5954d1eed5b1c37
I20260923 22:17:05.826512 3073 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e629a3d8ee14412a5954d1eed5b1c37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:05 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:38682
I20260923 22:17:05.826607 3189 tablet_replica.cc:330] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: stopping tablet replica
I20260923 22:17:05.826593 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e629a3d8ee14412a5954d1eed5b1c37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:05 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:34448
I20260923 22:17:05.826678 3189 raft_consensus.cc:2229] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:05.826712 3191 tablet_replica.cc:330] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: stopping tablet replica
I20260923 22:17:05.826761 3189 raft_consensus.cc:2258] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:05.826776 3191 raft_consensus.cc:2229] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:05.826813 3191 raft_consensus.cc:2258] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:05.826805 2933 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e629a3d8ee14412a5954d1eed5b1c37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:05 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:35940
I20260923 22:17:05.826903 3190 tablet_replica.cc:330] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: stopping tablet replica
I20260923 22:17:05.826967 3190 raft_consensus.cc:2229] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:05.827001 3190 raft_consensus.cc:2258] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:05.827149 3191 ts_tablet_manager.cc:1905] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:05.827673 3190 ts_tablet_manager.cc:1905] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:05.828967 3191 ts_tablet_manager.cc:1918] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:05.828974 3189 ts_tablet_manager.cc:1905] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:05.829023 3191 log.cc:1196] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/4e629a3d8ee14412a5954d1eed5b1c37
I20260923 22:17:05.829152 3190 ts_tablet_manager.cc:1918] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:05.829213 3190 log.cc:1196] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/4e629a3d8ee14412a5954d1eed5b1c37
I20260923 22:17:05.829252 3191 ts_tablet_manager.cc:1939] T 4e629a3d8ee14412a5954d1eed5b1c37 P 889981b69b094ec0a9d670d3e18a3105: Deleting consensus metadata
I20260923 22:17:05.829425 3190 ts_tablet_manager.cc:1939] T 4e629a3d8ee14412a5954d1eed5b1c37 P 41182953edad4ff79696003f0e7c280b: Deleting consensus metadata
I20260923 22:17:05.829596 2642 catalog_manager.cc:4841] TS 889981b69b094ec0a9d670d3e18a3105 (127.1.129.65:39457): tablet 4e629a3d8ee14412a5954d1eed5b1c37 (table test-table [id=8ef74389d0224ece97d9ea9f92dcc0a5]) successfully deleted
I20260923 22:17:05.830049 2642 catalog_manager.cc:4841] TS 41182953edad4ff79696003f0e7c280b (127.1.129.66:34299): tablet 4e629a3d8ee14412a5954d1eed5b1c37 (table test-table [id=8ef74389d0224ece97d9ea9f92dcc0a5]) successfully deleted
I20260923 22:17:05.830607 3189 ts_tablet_manager.cc:1918] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:05.830687 3189 log.cc:1196] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/4e629a3d8ee14412a5954d1eed5b1c37
I20260923 22:17:05.830897 3189 ts_tablet_manager.cc:1939] T 4e629a3d8ee14412a5954d1eed5b1c37 P 3c3f7ce07cf7411f95d935839021f4b0: Deleting consensus metadata
I20260923 22:17:05.831209 2640 catalog_manager.cc:4841] TS 3c3f7ce07cf7411f95d935839021f4b0 (127.1.129.67:44629): tablet 4e629a3d8ee14412a5954d1eed5b1c37 (table test-table [id=8ef74389d0224ece97d9ea9f92dcc0a5]) successfully deleted
Sep 23 22:17:05 dist-test-slave-dmw7 krb5kdc[2611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:05.845285 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.834944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.1.129.65:39457)
0923 22:17:05.834989 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.834991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.835142 (+ 151us) server_negotiation.cc:407] Connection header received
0923 22:17:05.835201 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.835203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.835215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.835276 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:05.835951 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.836712 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.837809 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.838114 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.838196 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.841796 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:05.841809 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:05.841809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:05.841818 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:05.843884 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:05.844325 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:05.844329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:05.844329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:05.844424 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:05.844657 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:05.844658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:05.844658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:05.844861 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:05.845017 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.845095 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.845183 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 22:17:05.845788 2853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34480
I20260923 22:17:05.854485 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:05.846786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.1.129.126:45323)
0923 22:17:05.846832 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:05.846834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:05.847057 (+ 223us) server_negotiation.cc:407] Connection header received
0923 22:17:05.847110 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:05.847111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:05.847125 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:05.847187 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:05.847831 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.848632 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.849667 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:05.850004 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:05.850046 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:05.851558 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:05.851559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:05.851560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:05.851567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:05.853207 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:05.853581 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:05.853582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:05.853582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:05.853631 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:05.853898 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:05.853899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:05.853899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:05.854015 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:05.854087 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:05.854301 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 22:17:05.854392 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 22:17:05.855629 2656 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56356
I20260923 22:17:05.856645 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 2728
I20260923 22:17:05.866279 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 2868
I20260923 22:17:05.872066 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 3008
I20260923 22:17:05.877170 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 2624
2026-09-23T22:17:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2709 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:05 dist-test-slave-dmw7 krb5kdc[3233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:05 dist-test-slave-dmw7 krb5kdc[3233](info): set up 2 sockets
Sep 23 22:17:05 dist-test-slave-dmw7 krb5kdc[3233](info): commencing operation
krb5kdc: starting...
W20260923 22:17:06.914229 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:06 dist-test-slave-dmw7 krb5kdc[3233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:07.028761 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34521
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:34521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:07.107740 3246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:07.108017 3246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:07.108062 3246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:07.109231 3246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:07.109302 3246 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 22:17:07.109324 3246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:07.109346 3246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:07.109375 3246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:07.109395 3246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:07.110906 3246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:34521
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:07.111200 3246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:07.111441 3246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:07.116238 3251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:07.116231 3254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:07.116410 3246 server_base.cc:1034] running on GCE node
W20260923 22:17:07.116231 3252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:07.116688 3246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:07.116999 3246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:07.118160 3246 hybrid_clock.cc:648] HybridClock initialized: now 1790201827118150 us; error 31 us; skew 500 ppm
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:07.120532 3246 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:07.120981 3246 webserver.cc:460] Webserver started at http://127.1.129.126:34355/ using document root <none> and password file <none>
I20260923 22:17:07.121204 3246 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:07.121279 3246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:07.121419 3246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:07.122453 3246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "3c17139bc6a34e549474f2f9d17ac779"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "48d7f5bf0bfdba01a95789f58c343a19"
server_key_iv: "d721b5ca125326c4e576f5570e272829"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.122833 3246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "3c17139bc6a34e549474f2f9d17ac779"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "48d7f5bf0bfdba01a95789f58c343a19"
server_key_iv: "d721b5ca125326c4e576f5570e272829"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.124117 3246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:07.124799 3261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.125023 3246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.125124 3246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "3c17139bc6a34e549474f2f9d17ac779"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "48d7f5bf0bfdba01a95789f58c343a19"
server_key_iv: "d721b5ca125326c4e576f5570e272829"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.125219 3246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:07.138886 3246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:07.141280 3246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:07.141441 3246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:07.145911 3246 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:34521
I20260923 22:17:07.145991 3323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:34521 every 8 connection(s)
I20260923 22:17:07.146318 3246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:07.147042 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:07.149248 3324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: Bootstrap starting.
I20260923 22:17:07.149824 3324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:07.150143 3324 log.cc:824] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:07.150828 3324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: No bootstrap required, opened a new log
I20260923 22:17:07.152875 3324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c17139bc6a34e549474f2f9d17ac779" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34521 } }
I20260923 22:17:07.153026 3324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:07.153049 3324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c17139bc6a34e549474f2f9d17ac779, State: Initialized, Role: FOLLOWER
I20260923 22:17:07.153218 3324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c17139bc6a34e549474f2f9d17ac779" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34521 } }
I20260923 22:17:07.153283 3324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:07.153306 3324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:07.153359 3324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:07.153541 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 3246
I20260923 22:17:07.153636 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:07.153798 1541 external_mini_cluster.cc:1428] Setting key 62fddf9521d7902b837da3dfa61e1033
I20260923 22:17:07.153990 3324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c17139bc6a34e549474f2f9d17ac779" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34521 } }
I20260923 22:17:07.154196 3324 leader_election.cc:302] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c17139bc6a34e549474f2f9d17ac779; no voters:
I20260923 22:17:07.154377 3324 leader_election.cc:288] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:07.154670 3327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:07.154681 3324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:07.154826 3327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [term 1 LEADER]: Becoming Leader. State: Replica: 3c17139bc6a34e549474f2f9d17ac779, State: Running, Role: LEADER
I20260923 22:17:07.154917 3327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c17139bc6a34e549474f2f9d17ac779" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34521 } }
I20260923 22:17:07.155321 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c17139bc6a34e549474f2f9d17ac779. Latest consensus state: current_term: 1 leader_uuid: "3c17139bc6a34e549474f2f9d17ac779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c17139bc6a34e549474f2f9d17ac779" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34521 } } }
I20260923 22:17:07.155423 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:07.155575 3329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c17139bc6a34e549474f2f9d17ac779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c17139bc6a34e549474f2f9d17ac779" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34521 } } }
I20260923 22:17:07.155632 3329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779 [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:07.156584 3344 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:07.156652 3344 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:07.156720 3338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:07.156890 3338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:07.158262 3338 catalog_manager.cc:1347] Generated new cluster ID: 6fcfa22ef8bd48448928bf1551e00d80
I20260923 22:17:07.158313 3338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:07.165868 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.154479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.1.129.126:34521)
0923 22:17:07.154739 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.154840 (+ 101us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.154926 (+ 86us) server_negotiation.cc:407] Connection header received
0923 22:17:07.155124 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.155132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.155214 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.155364 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.156093 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.157638 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.158726 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.159339 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.159455 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.162225 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.162229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.162229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.162241 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.164192 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.164586 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.164589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.164589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.164700 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.164909 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.164910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.164910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.165192 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.165336 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.165449 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.165656 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260923 22:17:07.167642 3338 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:07.168154 3338 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:07.174568 3338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: Generated new TSK 0
I20260923 22:17:07.174755 3338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:07.203594 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34521
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:07.283310 3350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:07.283546 3350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:07.283586 3350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:07.284731 3350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:07.284799 3350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:07.284833 3350 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 22:17:07.284865 3350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:07.284895 3350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:07.286350 3350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:34521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:07.286651 3350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:07.286885 3350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:07.292431 3355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:07.292352 3356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:07.292352 3358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:07.292405 3350 server_base.cc:1034] running on GCE node
I20260923 22:17:07.292833 3350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:07.293074 3350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:07.294250 3350 hybrid_clock.cc:648] HybridClock initialized: now 1790201827294234 us; error 32 us; skew 500 ppm
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:07.296936 3350 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:07.297441 3350 webserver.cc:460] Webserver started at http://127.1.129.65:44277/ using document root <none> and password file <none>
I20260923 22:17:07.297675 3350 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:07.297722 3350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:07.297873 3350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:07.299046 3350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "6d8d9536a7f3449a84f49f5094c0bcec"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "c22c788c05794446adcc65130e857466"
server_key_iv: "9742c0d96033dd0b5e3a4afa16f15b0a"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.299400 3350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "6d8d9536a7f3449a84f49f5094c0bcec"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "c22c788c05794446adcc65130e857466"
server_key_iv: "9742c0d96033dd0b5e3a4afa16f15b0a"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.300733 3350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:07.301481 3365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.301661 3350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:07.301731 3350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "6d8d9536a7f3449a84f49f5094c0bcec"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "c22c788c05794446adcc65130e857466"
server_key_iv: "9742c0d96033dd0b5e3a4afa16f15b0a"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.301821 3350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:07.355597 3350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:07.357632 3350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:07.357781 3350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:07.358356 3350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:07.358430 3350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.358471 3350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:07.358498 3350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.365830 3350 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:37059
I20260923 22:17:07.365926 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:37059 every 8 connection(s)
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:07.366294 3350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:07.368733 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 3350
I20260923 22:17:07.368836 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:07.369036 1541 external_mini_cluster.cc:1428] Setting key e80652a62f536e6c87e64f3924af5e4c
I20260923 22:17:07.369822 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.359026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:50147 (local address 127.1.129.126:34521)
0923 22:17:07.359105 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.359108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.359375 (+ 267us) server_negotiation.cc:407] Connection header received
0923 22:17:07.360164 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.360166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.360178 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.360246 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.361234 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.362087 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.363849 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.364058 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.364117 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.366674 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.366675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.366676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.366686 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.368190 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.368619 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.368620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.368620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.368685 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.368997 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.368998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.368999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.369147 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.369219 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.369581 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.369695 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 22:17:07.370018 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.359014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34521 (local address 127.1.129.65:50147)
0923 22:17:07.359220 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.359271 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.360029 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.360362 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.360367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.360800 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:07.361121 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.361129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.362193 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.362194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.363742 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.363745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.364149 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.364150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.364316 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.364946 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:07.364954 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:07.366523 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:07.368283 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.368289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.368291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.368542 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.368757 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.368757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.368757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.368935 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.369272 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:07.369279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:07.369444 (+ 165us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.369534 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.369741 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260923 22:17:07.370697 3481 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34521
I20260923 22:17:07.370795 3481 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:07.370990 3481 heartbeater.cc:511] Master 127.1.129.126:34521 requested a full tablet report, sending...
I20260923 22:17:07.371543 3277 ts_manager.cc:194] Registered new tserver with Master: 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059)
I20260923 22:17:07.372174 3277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:50147
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
I20260923 22:17:07.378309 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.372620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:41809 (local address 127.1.129.126:34521)
0923 22:17:07.372665 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.372666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.372819 (+ 153us) server_negotiation.cc:407] Connection header received
0923 22:17:07.372859 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.372860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.372870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.372918 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.373226 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.373729 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.374428 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.374673 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.374732 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.375755 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.375757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.375757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.375764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.377279 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.377611 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.377612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.377613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.377703 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.377915 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.377915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.377916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.378015 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.378069 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.378177 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.378249 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:07.378314 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.372579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34521 (local address 127.1.129.65:41809)
0923 22:17:07.372739 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.372750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.372807 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.372919 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.372920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.373106 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:07.373154 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.373155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.373826 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.373827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.374333 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.374334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.374729 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.374730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.374801 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.375270 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:07.375275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:07.375589 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:07.377365 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.377366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.377366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.377538 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.377786 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.377786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.377787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.377845 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.378071 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:07.378072 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:07.378109 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.378160 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.378238 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:07.394711 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34521
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:07.475586 3490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:07.475788 3490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:07.475814 3490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:07.476881 3490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:07.476915 3490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:07.476935 3490 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 22:17:07.476948 3490 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:07.476964 3490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:07.478312 3490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:34521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:07.478505 3490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:07.478705 3490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:07.483793 3496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:07.483922 3490 server_base.cc:1034] running on GCE node
W20260923 22:17:07.483819 3498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:07.483811 3495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:07.484292 3490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:07.484542 3490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:07.485719 3490 hybrid_clock.cc:648] HybridClock initialized: now 1790201827485693 us; error 40 us; skew 500 ppm
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:07.488306 3490 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:07.488797 3490 webserver.cc:460] Webserver started at http://127.1.129.66:37429/ using document root <none> and password file <none>
I20260923 22:17:07.489019 3490 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:07.489060 3490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:07.489147 3490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:07.490152 3490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "b8e2b107e43649c8b9cf87ef6f6e859d"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "d08712f93fd73fa5806043a3b8848d28"
server_key_iv: "8c06b23d5d88bfbf89f0fc98f4566d26"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.490450 3490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "b8e2b107e43649c8b9cf87ef6f6e859d"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "d08712f93fd73fa5806043a3b8848d28"
server_key_iv: "8c06b23d5d88bfbf89f0fc98f4566d26"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.491766 3490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:07.492465 3505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.492616 3490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:07.492705 3490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "b8e2b107e43649c8b9cf87ef6f6e859d"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "d08712f93fd73fa5806043a3b8848d28"
server_key_iv: "8c06b23d5d88bfbf89f0fc98f4566d26"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.492759 3490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:07.516115 3490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:07.518532 3490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:07.518708 3490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:07.519197 3490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:07.519232 3490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.519255 3490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:07.519330 3490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.526516 3490 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:42113
I20260923 22:17:07.526666 3620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:42113 every 8 connection(s)
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:07.526988 3490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:07.529551 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 3490
I20260923 22:17:07.529649 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:07.529821 1541 external_mini_cluster.cc:1428] Setting key faad38d315fd158faa4a698992aea702
I20260923 22:17:07.530280 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.519765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:32973 (local address 127.1.129.126:34521)
0923 22:17:07.519843 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.519846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.520029 (+ 183us) server_negotiation.cc:407] Connection header received
0923 22:17:07.520518 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.520520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.520534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.520620 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.521596 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.522392 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.524140 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.524483 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.524554 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.527512 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.527513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.527514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.527522 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.528770 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.529259 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.529261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.529262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.529326 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.529572 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.529573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.529573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.529719 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.529781 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.530103 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.530182 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 22:17:07.530511 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.519724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34521 (local address 127.1.129.66:32973)
0923 22:17:07.519892 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.519934 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.520416 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.520725 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.520729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.521119 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:07.521504 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.521511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.522498 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.522500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.524026 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.524029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.524570 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.524571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.524732 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.525395 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:07.525406 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:07.527391 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:07.528865 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.528872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.528874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.529164 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.529392 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.529392 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.529393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.529505 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.529772 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:07.529778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:07.529952 (+ 174us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.530081 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.530252 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260923 22:17:07.531008 3621 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34521
I20260923 22:17:07.531101 3621 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:07.531301 3621 heartbeater.cc:511] Master 127.1.129.126:34521 requested a full tablet report, sending...
I20260923 22:17:07.531630 3277 ts_manager.cc:194] Registered new tserver with Master: b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113)
I20260923 22:17:07.532130 3277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:32973
I20260923 22:17:07.538050 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.532113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34521 (local address 127.1.129.66:53979)
0923 22:17:07.532260 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.532270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.532329 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.532448 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.532448 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.532647 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:07.532715 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.532717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.533437 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.533437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.533928 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.533929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.534392 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.534394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.534458 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.534949 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:07.534961 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:07.535306 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:07.537068 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.537069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.537069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.537258 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.537475 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.537475 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.537476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.537520 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.537829 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:07.537830 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:07.537865 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.537911 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.537987 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":110,"threads_started":1}
I20260923 22:17:07.538120 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.532032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:53979 (local address 127.1.129.126:34521)
0923 22:17:07.532082 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.532084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.532336 (+ 252us) server_negotiation.cc:407] Connection header received
0923 22:17:07.532381 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.532382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.532392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.532435 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.532807 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.533345 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.534010 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.534314 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.534382 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.535397 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.535398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.535399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.535406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.536973 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.537336 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.537337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.537337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.537399 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.537577 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.537578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.537578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.537764 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.537833 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.537946 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.538025 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:07.558305 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34521
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:07.638842 3630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:07.639078 3630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:07.639120 3630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:07.640288 3630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:07.640357 3630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:07.640396 3630 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 22:17:07.640443 3630 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:07.640483 3630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:07.641952 3630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:34521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:07.642213 3630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:07.642443 3630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:07.647362 3638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:07.647375 3635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:07.647377 3636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:07.647576 3630 server_base.cc:1034] running on GCE node
I20260923 22:17:07.647785 3630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:07.648059 3630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:07.649219 3630 hybrid_clock.cc:648] HybridClock initialized: now 1790201827649209 us; error 26 us; skew 500 ppm
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:07.651669 3630 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:07.652084 3630 webserver.cc:460] Webserver started at http://127.1.129.67:36797/ using document root <none> and password file <none>
I20260923 22:17:07.652324 3630 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:07.652393 3630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:07.652546 3630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:07.653615 3630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "0155cc3683184257ba80c730dc2d87fc"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "3d06a54f972aa52d21565b3f94bae4c0"
server_key_iv: "117a543a183a3bd54d8a595a54e30563"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.653909 3630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "0155cc3683184257ba80c730dc2d87fc"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "3d06a54f972aa52d21565b3f94bae4c0"
server_key_iv: "117a543a183a3bd54d8a595a54e30563"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.655431 3630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:07.656323 3645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.656504 3630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:07.656575 3630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "0155cc3683184257ba80c730dc2d87fc"
format_stamp: "Formatted at 2026-09-23 22:17:07 on dist-test-slave-dmw7"
server_key: "3d06a54f972aa52d21565b3f94bae4c0"
server_key_iv: "117a543a183a3bd54d8a595a54e30563"
server_key_version: "encryptionkey@0"
I20260923 22:17:07.656627 3630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:07.675303 3630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:07.677183 3630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:07.677304 3630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:07.677734 3630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:07.677769 3630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.677801 3630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:07.677816 3630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.684500 3630 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:34859
I20260923 22:17:07.684530 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:34859 every 8 connection(s)
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:07.684888 3630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:07.687744 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.678298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:60707 (local address 127.1.129.126:34521)
0923 22:17:07.678352 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.678354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.678554 (+ 200us) server_negotiation.cc:407] Connection header received
0923 22:17:07.679120 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.679121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.679134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.679181 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.680029 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.680809 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.682363 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.682704 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.682777 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.685200 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.685202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.685202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.685211 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.686723 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.687054 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.687055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.687055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.687104 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.687294 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.687294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.687295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.687387 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.687443 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.687629 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.687684 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:17:07.687916 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.678283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34521 (local address 127.1.129.67:60707)
0923 22:17:07.678437 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.678471 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.679003 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.679273 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.679281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.679640 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:07.679945 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.679952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.680903 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.680905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.682242 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.682246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.682793 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.682794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.682926 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.683473 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:07.683483 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:07.685116 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:07.686795 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.686799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.686800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.686993 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.687157 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.687158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.687158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.687241 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.687438 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:07.687442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:07.687542 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.687618 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.687725 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260923 22:17:07.688411 3761 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34521
I20260923 22:17:07.688501 3761 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:07.688681 3761 heartbeater.cc:511] Master 127.1.129.126:34521 requested a full tablet report, sending...
I20260923 22:17:07.689117 3278 ts_manager.cc:194] Registered new tserver with Master: 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:07.689568 3278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:60707
I20260923 22:17:07.693167 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 3630
I20260923 22:17:07.693259 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:07.693432 1541 external_mini_cluster.cc:1428] Setting key 172c8f65bd008f070b7c7115be90ceea
I20260923 22:17:07.694113 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:07.694947 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.689434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:41859 (local address 127.1.129.126:34521)
0923 22:17:07.689479 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.689481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.689603 (+ 122us) server_negotiation.cc:407] Connection header received
0923 22:17:07.689672 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.689673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.689683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.689737 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.690053 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.690503 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.691230 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.691420 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.691502 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.692385 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.692386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.692386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.692392 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.693836 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.694179 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.694180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.694180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.694232 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.694432 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.694433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.694433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.694539 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.694614 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.694794 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.694859 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:07.694963 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.689381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34521 (local address 127.1.129.67:41859)
0923 22:17:07.689527 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.689536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.689615 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.689741 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.689742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.689924 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:07.689968 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.689969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.690598 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.690598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.691132 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.691133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.691504 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.691505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.691577 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.692001 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:07.692006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:07.692269 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:07.693930 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.693931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.693931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.694095 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.694305 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.694306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.694306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.694361 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.694607 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:07.694608 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:07.694672 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.694762 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.694861 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:07.716596 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.709929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.1.129.126:34521)
0923 22:17:07.709984 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.709986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.709998 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:07.710062 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.710063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.710071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.710121 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.710719 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.711202 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.711924 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.712133 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.712193 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.714492 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.714493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.714493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.714499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.715724 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.716012 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.716013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.716013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.716062 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.716203 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.716203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.716203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.716292 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.716362 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.716446 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.716503 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:17:07.718243 3278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:07.719875 3278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:07.726130 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.722132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:34859 (local address 127.0.0.1:54156)
0923 22:17:07.722398 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.722408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.722460 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.722686 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.722689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.722692 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:07.722765 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.722767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.723837 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.723838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.724800 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.724801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.725791 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.725792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.725884 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.725894 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.725985 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.726039 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":86,"threads_started":1}
I20260923 22:17:07.726338 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.722291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.1.129.67:34859)
0923 22:17:07.722482 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.722487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.722507 (+ 20us) server_negotiation.cc:407] Connection header received
0923 22:17:07.722551 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.722552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.722563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.722618 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:07.722840 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.723755 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.724895 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.725616 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.725677 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.725734 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.726004 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.726041 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260923 22:17:07.726624 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.722320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:37059 (local address 127.0.0.1:46112)
0923 22:17:07.722731 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.722742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.722869 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.723249 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.723250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.723252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:07.723300 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.723302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.724520 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.724521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.725408 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.725411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.726386 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.726387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.726447 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.726452 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.726501 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.726540 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":65,"threads_started":1}
I20260923 22:17:07.726818 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.722128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:42113 (local address 127.0.0.1:37358)
0923 22:17:07.722273 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.722314 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.722385 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.723196 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.723197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.723199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:07.723238 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.723239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.724510 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.724511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.725595 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.725598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.726527 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.726528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.726600 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.726606 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.726703 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.726752 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":25,"thread_start_us":52,"threads_started":1}
I20260923 22:17:07.726948 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.722523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.1.129.65:37059)
0923 22:17:07.722728 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.722732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.722822 (+ 90us) server_negotiation.cc:407] Connection header received
0923 22:17:07.723004 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.723006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.723080 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.723187 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:07.723363 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.724434 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.725494 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.726223 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.726295 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.726349 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.726820 (+ 471us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.726860 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":120,"threads_started":1}
I20260923 22:17:07.727144 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.722232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.1.129.66:42113)
0923 22:17:07.723001 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.723005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.723015 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:07.723057 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.723059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.723075 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.723186 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:07.723292 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.724405 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.725686 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.726420 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.726518 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.726572 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.726996 (+ 424us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.727050 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":84,"threads_started":1}
I20260923 22:17:07.727236 3690 tablet_service.cc:1467] Processing CreateTablet for tablet e62e66d0fb3542639d0e81c2533d462e (DEFAULT_TABLE table=test-table [id=e70ebaa44ef344b98d02bb100069f491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:07.727577 3415 tablet_service.cc:1467] Processing CreateTablet for tablet e62e66d0fb3542639d0e81c2533d462e (DEFAULT_TABLE table=test-table [id=e70ebaa44ef344b98d02bb100069f491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:07.727662 3690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e66d0fb3542639d0e81c2533d462e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:07.727843 3415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e66d0fb3542639d0e81c2533d462e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:07.730343 3555 tablet_service.cc:1467] Processing CreateTablet for tablet e62e66d0fb3542639d0e81c2533d462e (DEFAULT_TABLE table=test-table [id=e70ebaa44ef344b98d02bb100069f491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:07.730618 3784 tablet_bootstrap.cc:492] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: Bootstrap starting.
I20260923 22:17:07.730700 3555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e66d0fb3542639d0e81c2533d462e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:07.730952 3785 tablet_bootstrap.cc:492] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Bootstrap starting.
I20260923 22:17:07.731626 3784 tablet_bootstrap.cc:654] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:07.731628 3785 tablet_bootstrap.cc:654] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:07.731951 3784 log.cc:824] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:07.731988 3785 log.cc:824] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:07.732728 3784 tablet_bootstrap.cc:492] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: No bootstrap required, opened a new log
I20260923 22:17:07.732817 3784 ts_tablet_manager.cc:1397] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:17:07.732946 3785 tablet_bootstrap.cc:492] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: No bootstrap required, opened a new log
I20260923 22:17:07.733028 3785 ts_tablet_manager.cc:1397] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:07.734280 3784 raft_consensus.cc:348] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.734395 3788 tablet_bootstrap.cc:492] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Bootstrap starting.
I20260923 22:17:07.734462 3784 raft_consensus.cc:374] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:07.734503 3784 raft_consensus.cc:729] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0155cc3683184257ba80c730dc2d87fc, State: Initialized, Role: FOLLOWER
I20260923 22:17:07.734656 3784 consensus_queue.cc:260] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.734927 3784 ts_tablet_manager.cc:1428] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:07.735059 3761 heartbeater.cc:503] Master 127.1.129.126:34521 was elected leader, sending a full tablet report...
I20260923 22:17:07.735188 3785 raft_consensus.cc:348] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.735337 3785 raft_consensus.cc:374] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:07.735379 3785 raft_consensus.cc:729] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8d9536a7f3449a84f49f5094c0bcec, State: Initialized, Role: FOLLOWER
I20260923 22:17:07.735495 3785 consensus_queue.cc:260] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.735600 3788 tablet_bootstrap.cc:654] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:07.735778 3785 ts_tablet_manager.cc:1428] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:17:07.735795 3481 heartbeater.cc:503] Master 127.1.129.126:34521 was elected leader, sending a full tablet report...
I20260923 22:17:07.735956 3788 log.cc:824] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:07.736718 3788 tablet_bootstrap.cc:492] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: No bootstrap required, opened a new log
I20260923 22:17:07.736801 3788 ts_tablet_manager.cc:1397] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:07.738214 3788 raft_consensus.cc:348] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.738373 3788 raft_consensus.cc:374] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:07.738436 3788 raft_consensus.cc:729] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e2b107e43649c8b9cf87ef6f6e859d, State: Initialized, Role: FOLLOWER
I20260923 22:17:07.738549 3788 consensus_queue.cc:260] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.738844 3788 ts_tablet_manager.cc:1428] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:07.738886 3621 heartbeater.cc:503] Master 127.1.129.126:34521 was elected leader, sending a full tablet report...
W20260923 22:17:07.777580 3622 tablet.cc:2369] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:07.854851 3790 raft_consensus.cc:482] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:07.855051 3790 raft_consensus.cc:504] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.855558 3790 leader_election.cc:288] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:07.859227 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.855905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:37437 (local address 127.1.129.66:42113)
0923 22:17:07.855979 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.855981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.856006 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:17:07.856052 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.856053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.856065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.856131 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:07.856670 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.857458 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.858348 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.858809 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.858925 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.858935 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.859119 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.859159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:17:07.859228 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.855802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:42113 (local address 127.1.129.65:37437)
0923 22:17:07.855866 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.855879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.855928 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.856462 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.856464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.856466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:07.856546 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.856548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.857545 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.857546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.858264 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.858265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.858919 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.858920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.859010 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.859014 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.859109 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.859153 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 22:17:07.859630 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e66d0fb3542639d0e81c2533d462e" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" is_pre_election: true
I20260923 22:17:07.859807 3575 raft_consensus.cc:2454] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 0.
I20260923 22:17:07.860054 3368 leader_election.cc:302] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8d9536a7f3449a84f49f5094c0bcec, b8e2b107e43649c8b9cf87ef6f6e859d; no voters:
I20260923 22:17:07.860224 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.856314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:59737 (local address 127.1.129.67:34859)
0923 22:17:07.856398 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.856400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.856408 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:07.856469 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.856469 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.856477 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.856594 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:07.856781 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.857474 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.859200 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.859811 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.859887 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.859895 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.860113 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.860156 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 22:17:07.860215 3790 raft_consensus.cc:2781] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:07.860282 3790 raft_consensus.cc:482] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:07.860324 3790 raft_consensus.cc:3037] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:07.860399 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.855811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:34859 (local address 127.1.129.65:59737)
0923 22:17:07.856309 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.856320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.856379 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.856631 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.856632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.856634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:07.856710 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.856712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.858345 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.858346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.859115 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.859117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.859917 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.859919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.859993 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.859998 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.860099 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.860196 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":14,"thread_start_us":75,"threads_started":1}
I20260923 22:17:07.860785 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e66d0fb3542639d0e81c2533d462e" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc" is_pre_election: true
I20260923 22:17:07.860905 3715 raft_consensus.cc:2454] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 0.
I20260923 22:17:07.861244 3790 raft_consensus.cc:504] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.861388 3790 leader_election.cc:288] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 election: Requested vote from peers b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:07.861541 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e66d0fb3542639d0e81c2533d462e" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc"
I20260923 22:17:07.861575 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e66d0fb3542639d0e81c2533d462e" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d"
I20260923 22:17:07.861624 3715 raft_consensus.cc:3037] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:07.861647 3575 raft_consensus.cc:3037] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:07.862327 3575 raft_consensus.cc:2454] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 1.
I20260923 22:17:07.862334 3715 raft_consensus.cc:2454] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 1.
I20260923 22:17:07.862514 3368 leader_election.cc:302] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8d9536a7f3449a84f49f5094c0bcec, b8e2b107e43649c8b9cf87ef6f6e859d; no voters:
I20260923 22:17:07.862605 3790 raft_consensus.cc:2781] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:07.862780 3790 raft_consensus.cc:686] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 LEADER]: Becoming Leader. State: Replica: 6d8d9536a7f3449a84f49f5094c0bcec, State: Running, Role: LEADER
I20260923 22:17:07.862876 3790 consensus_queue.cc:237] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } }
I20260923 22:17:07.863652 3277 catalog_manager.cc:5495] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65). New cstate: current_term: 1 leader_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } health_report { overall_health: HEALTHY } } }
W20260923 22:17:07.867023 3482 tablet.cc:2369] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 22:17:07 dist-test-slave-dmw7 krb5kdc[3233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:07.903594 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.891256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.1.129.126:34521)
0923 22:17:07.891331 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:07.891333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:07.891441 (+ 108us) server_negotiation.cc:407] Connection header received
0923 22:17:07.891542 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:07.891543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:07.891553 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:07.891624 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:07.892112 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.892671 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.893380 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.893560 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.893633 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.901085 (+ 7452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:07.901087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:07.901088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:07.901095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:07.902433 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.902904 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.902905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.902905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.902956 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:07.903155 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:07.903156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:07.903156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:07.903255 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:07.903303 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:07.903463 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 22:17:07.903527 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:17:07.903759 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:07.891211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34521 (local address 127.0.0.1:58002)
0923 22:17:07.891361 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:07.891375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:07.891456 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:07.891629 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:07.891630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:07.891971 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:07.892021 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.892024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.892758 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.892759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.893294 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:07.893296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:07.893640 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:07.893641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:07.893700 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:07.899595 (+ 5895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:07.899601 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:07.900969 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:07.902517 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.902524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.902526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.902827 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.903019 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:07.903020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:07.903020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:07.903083 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:07.903328 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:07.903334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:07.903380 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 22:17:07.903446 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:17:07.903593 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":55,"thread_start_us":69,"threads_started":1}
I20260923 22:17:07.905824 3277 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:58002:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 22:17:07.906030 3277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:07.907572 3690 tablet_service.cc:1467] Processing CreateTablet for tablet 00b3a8d3ecb74a888be145dafba39a9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=57981eba00184ea3b782e01379536b98]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:07.907553 3415 tablet_service.cc:1467] Processing CreateTablet for tablet 00b3a8d3ecb74a888be145dafba39a9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=57981eba00184ea3b782e01379536b98]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:07.907595 3555 tablet_service.cc:1467] Processing CreateTablet for tablet 00b3a8d3ecb74a888be145dafba39a9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=57981eba00184ea3b782e01379536b98]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:07.907739 3415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b3a8d3ecb74a888be145dafba39a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:07.907739 3555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b3a8d3ecb74a888be145dafba39a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:07.907740 3690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b3a8d3ecb74a888be145dafba39a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:07.908931 3784 tablet_bootstrap.cc:492] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: Bootstrap starting.
I20260923 22:17:07.909267 3788 tablet_bootstrap.cc:492] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: Bootstrap starting.
I20260923 22:17:07.909503 3784 tablet_bootstrap.cc:654] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:07.909897 3788 tablet_bootstrap.cc:654] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:07.910104 3784 tablet_bootstrap.cc:492] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: No bootstrap required, opened a new log
I20260923 22:17:07.910145 3784 ts_tablet_manager.cc:1397] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:07.910274 3785 tablet_bootstrap.cc:492] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: Bootstrap starting.
I20260923 22:17:07.910305 3784 raft_consensus.cc:348] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:07.910406 3784 raft_consensus.cc:374] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:07.910439 3784 raft_consensus.cc:729] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0155cc3683184257ba80c730dc2d87fc, State: Initialized, Role: FOLLOWER
I20260923 22:17:07.910491 3784 consensus_queue.cc:260] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:07.910585 3784 ts_tablet_manager.cc:1428] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:07.910718 3789 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:07.910826 3789 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:07.910950 3785 tablet_bootstrap.cc:654] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:07.911515 3788 tablet_bootstrap.cc:492] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: No bootstrap required, opened a new log
I20260923 22:17:07.911569 3788 ts_tablet_manager.cc:1397] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:07.911610 3785 tablet_bootstrap.cc:492] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: No bootstrap required, opened a new log
I20260923 22:17:07.911660 3785 ts_tablet_manager.cc:1397] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:07.911727 3788 raft_consensus.cc:348] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:07.911799 3788 raft_consensus.cc:374] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:07.911814 3788 raft_consensus.cc:729] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e2b107e43649c8b9cf87ef6f6e859d, State: Initialized, Role: FOLLOWER
I20260923 22:17:07.911820 3785 raft_consensus.cc:348] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:07.911868 3788 consensus_queue.cc:260] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:07.911899 3785 raft_consensus.cc:374] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:07.911931 3785 raft_consensus.cc:729] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8d9536a7f3449a84f49f5094c0bcec, State: Initialized, Role: FOLLOWER
I20260923 22:17:07.911950 3788 ts_tablet_manager.cc:1428] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:07.911983 3785 consensus_queue.cc:260] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:07.912075 3785 ts_tablet_manager.cc:1428] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:07.912072 3792 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:07.912164 3792 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:07.912199 3790 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:07.912266 3790 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 22:17:07.935397 3762 tablet.cc:2369] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:07.999480 3792 raft_consensus.cc:482] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:07.999567 3792 raft_consensus.cc:504] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:07.999989 3792 leader_election.cc:288] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:08.002871 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.000205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:37059 (local address 127.1.129.66:41031)
0923 22:17:08.000244 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:08.000256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:08.000299 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:08.000441 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:08.000442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:08.000444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:08.000509 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.000511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.001242 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.001243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.002020 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.002022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.002585 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.002586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.002720 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.002725 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:08.002780 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:17:08.002818 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 22:17:08.002916 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.000254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:41031 (local address 127.1.129.65:37059)
0923 22:17:08.000317 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:08.000319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:08.000330 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:08.000363 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:08.000364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:08.000371 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:08.000419 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:08.000588 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.001167 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.002102 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.002496 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.002549 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.002556 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:08.002799 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 22:17:08.002834 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:17:08.003070 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.000296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:47773 (local address 127.1.129.67:34859)
0923 22:17:08.000342 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:08.000344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:08.000571 (+ 227us) server_negotiation.cc:407] Connection header received
0923 22:17:08.000628 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:08.000629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:08.000646 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:08.000685 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:08.000829 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.001328 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.002236 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.002627 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.002740 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.002750 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:08.002945 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 22:17:08.002989 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:08.003070 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.000208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:34859 (local address 127.1.129.66:47773)
0923 22:17:08.000490 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:08.000500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:08.000573 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:08.000689 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:08.000690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:08.000691 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:08.000764 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.000766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.001408 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.001408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.002127 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.002129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.002796 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.002797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.002861 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.002866 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:08.002924 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:17:08.002964 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":144,"thread_start_us":76,"threads_started":1}
I20260923 22:17:08.003291 3435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b3a8d3ecb74a888be145dafba39a9f" candidate_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" is_pre_election: true
I20260923 22:17:08.003405 3435 raft_consensus.cc:2454] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e2b107e43649c8b9cf87ef6f6e859d in term 0.
I20260923 22:17:08.003398 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b3a8d3ecb74a888be145dafba39a9f" candidate_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc" is_pre_election: true
I20260923 22:17:08.003507 3715 raft_consensus.cc:2454] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e2b107e43649c8b9cf87ef6f6e859d in term 0.
I20260923 22:17:08.003687 3508 leader_election.cc:302] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8d9536a7f3449a84f49f5094c0bcec, b8e2b107e43649c8b9cf87ef6f6e859d; no voters:
I20260923 22:17:08.003827 3792 raft_consensus.cc:2781] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:08.003870 3792 raft_consensus.cc:482] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:08.003897 3792 raft_consensus.cc:3037] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:08.004575 3792 raft_consensus.cc:504] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.004726 3792 leader_election.cc:288] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [CANDIDATE]: Term 1 election: Requested vote from peers 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:08.004879 3435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b3a8d3ecb74a888be145dafba39a9f" candidate_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8d9536a7f3449a84f49f5094c0bcec"
I20260923 22:17:08.004891 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b3a8d3ecb74a888be145dafba39a9f" candidate_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc"
I20260923 22:17:08.004956 3435 raft_consensus.cc:3037] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:08.004983 3715 raft_consensus.cc:3037] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:08.005587 3715 raft_consensus.cc:2454] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e2b107e43649c8b9cf87ef6f6e859d in term 1.
I20260923 22:17:08.005676 3435 raft_consensus.cc:2454] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e2b107e43649c8b9cf87ef6f6e859d in term 1.
I20260923 22:17:08.005793 3506 leader_election.cc:302] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0155cc3683184257ba80c730dc2d87fc, b8e2b107e43649c8b9cf87ef6f6e859d; no voters:
I20260923 22:17:08.005882 3792 raft_consensus.cc:2781] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:08.006031 3792 raft_consensus.cc:686] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 LEADER]: Becoming Leader. State: Replica: b8e2b107e43649c8b9cf87ef6f6e859d, State: Running, Role: LEADER
I20260923 22:17:08.006135 3792 consensus_queue.cc:237] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.006546 3806 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: TxnStatusTablet state changed. Reason: New leader b8e2b107e43649c8b9cf87ef6f6e859d. Latest consensus state: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:08.006652 3806 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:08.006839 3808 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 22:17:08.006956 3808 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 22:17:08.006928 3277 catalog_manager.cc:5495] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d reported cstate change: term changed from 0 to 1, leader changed from <none> to b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66). New cstate: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:08.027704 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.024948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.1.129.66:42113)
0923 22:17:08.025012 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:08.025014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:08.025023 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:08.025058 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:08.025059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:08.025078 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:08.025160 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:08.025255 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.026013 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.026796 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.027069 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.027149 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.027338 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:08.027453 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:08.027535 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:08.027587 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 22:17:08.027607 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:17:08.027691 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.024852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:42113 (local address 127.0.0.1:37368)
0923 22:17:08.024926 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:08.024938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:08.024984 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:08.025159 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:08.025160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:08.025160 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:08.025204 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.025206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.026105 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.026105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.026712 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.026715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.027146 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.027147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.027214 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.027227 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:08.027524 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:08.027532 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 22:17:08.027573 (+ 41us) client_negotiation.cc:240] Negotiation successful
0923 22:17:08.027619 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 22:17:08.029455 3435 raft_consensus.cc:1261] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Refusing update from remote peer b8e2b107e43649c8b9cf87ef6f6e859d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:08.029464 3715 raft_consensus.cc:1261] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Refusing update from remote peer b8e2b107e43649c8b9cf87ef6f6e859d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:08.029798 3792 consensus_queue.cc:1035] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:08.029996 3806 consensus_queue.cc:1035] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:08.031119 3789 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: TxnStatusTablet state changed. Reason: New leader b8e2b107e43649c8b9cf87ef6f6e859d. Latest consensus state: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:08.031216 3789 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:08.031810 3790 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: TxnStatusTablet state changed. Reason: New leader b8e2b107e43649c8b9cf87ef6f6e859d. Latest consensus state: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:08.031909 3790 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:08.032779 3806 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:08.032874 3806 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:08.033440 3790 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:08.033521 3790 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P 6d8d9536a7f3449a84f49f5094c0bcec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:08.033957 3814 mvcc.cc:205] Tried to move back new op lower bound from 7332666687606210560 to 7332666687513722880. Current Snapshot: MvccSnapshot[applied={T|T < 7332666687606210560 or (T in {7332666687606210560})}]
I20260923 22:17:08.034561 3792 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:08.034665 3792 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P b8e2b107e43649c8b9cf87ef6f6e859d: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:08.035149 3812 mvcc.cc:205] Tried to move back new op lower bound from 7332666687606210560 to 7332666687513722880. Current Snapshot: MvccSnapshot[applied={T|T < 7332666687606210560}]
I20260923 22:17:08.035475 3789 tablet_replica.cc:439] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } } }
I20260923 22:17:08.035568 3789 tablet_replica.cc:442] T 00b3a8d3ecb74a888be145dafba39a9f P 0155cc3683184257ba80c730dc2d87fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:08.041685 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.039010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.1.129.65:37059)
0923 22:17:08.039045 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:08.039047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:08.039058 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:08.039136 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:08.039137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:08.039151 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:08.039202 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:08.039333 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.039781 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.040863 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.041039 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.041121 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.041296 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:08.041380 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:08.041439 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:08.041502 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 22:17:08.041519 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:17:08.046305 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.043938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:53851 (local address 127.1.129.66:42113)
0923 22:17:08.043976 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:08.043978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:08.044149 (+ 171us) server_negotiation.cc:407] Connection header received
0923 22:17:08.044232 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:08.044233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:08.044242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:08.044291 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:08.044410 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.044865 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.045614 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.045814 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.045885 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.046062 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:08.046099 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:08.046157 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:08.046218 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 22:17:08.046238 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:17:08.046350 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.043879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:42113 (local address 127.1.129.65:53851)
0923 22:17:08.044087 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:08.044097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:08.044174 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:08.044293 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:08.044294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:08.044295 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:08.044348 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.044349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.044949 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.044950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.045531 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.045533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.045884 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.045884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.045960 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.045972 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:08.046158 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:08.046160 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:08.046213 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:17:08.046252 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260923 22:17:08.048823 3715 raft_consensus.cc:1261] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Refusing update from remote peer 6d8d9536a7f3449a84f49f5094c0bcec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:08.048872 3575 raft_consensus.cc:1261] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Refusing update from remote peer 6d8d9536a7f3449a84f49f5094c0bcec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:08.049091 3794 consensus_queue.cc:1035] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:08.049211 3790 consensus_queue.cc:1035] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:08.061408 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.055793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:48533 (local address 127.1.129.65:37059)
0923 22:17:08.055827 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:08.055829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:08.058808 (+ 2979us) server_negotiation.cc:407] Connection header received
0923 22:17:08.058873 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:08.058873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:08.058881 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:08.058963 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:08.059106 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.059658 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.060822 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.060993 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.061046 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.061226 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:08.061249 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:08.061282 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:08.061346 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 22:17:08.061361 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:17:08.061476 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:08.055741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:37059 (local address 127.1.129.66:48533)
0923 22:17:08.058736 (+ 2995us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:08.058748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:08.058815 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:08.058968 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:08.058969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:08.058979 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:08.059048 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.059049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.059733 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.059734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.060737 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:08.060740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:08.061063 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:08.061064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:08.061131 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:08.061147 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:08.061295 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:08.061296 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:08.061345 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:17:08.061386 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2959,"thread_start_us":78,"threads_started":1}
I20260923 22:17:08.071113 3277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:08.071226 3277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:08.071939 3277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: Sending DeleteTablet for 3 replicas of tablet e62e66d0fb3542639d0e81c2533d462e
I20260923 22:17:08.072224 3690 tablet_service.cc:1514] Processing DeleteTablet for tablet e62e66d0fb3542639d0e81c2533d462e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:08 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:54156
I20260923 22:17:08.072299 3415 tablet_service.cc:1514] Processing DeleteTablet for tablet e62e66d0fb3542639d0e81c2533d462e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:08 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:46112
I20260923 22:17:08.072547 3831 tablet_replica.cc:330] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: stopping tablet replica
I20260923 22:17:08.072702 3831 raft_consensus.cc:2229] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:08.072768 3831 raft_consensus.cc:2258] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:08.072793 3277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:08.072935 3277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:08.073187 3831 ts_tablet_manager.cc:1905] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:08.073364 3555 tablet_service.cc:1514] Processing DeleteTablet for tablet e62e66d0fb3542639d0e81c2533d462e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:08 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:37358
I20260923 22:17:08.074229 3690 tablet_service.cc:1467] Processing CreateTablet for tablet ffe9b76ad18d478c83fc930f819979e9 (DEFAULT_TABLE table=test-table [id=072e11c4988446cd9ab5491b545c31ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:08.074353 3690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe9b76ad18d478c83fc930f819979e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:08.074891 3555 tablet_service.cc:1467] Processing CreateTablet for tablet ffe9b76ad18d478c83fc930f819979e9 (DEFAULT_TABLE table=test-table [id=072e11c4988446cd9ab5491b545c31ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:08.074895 3415 tablet_service.cc:1467] Processing CreateTablet for tablet ffe9b76ad18d478c83fc930f819979e9 (DEFAULT_TABLE table=test-table [id=072e11c4988446cd9ab5491b545c31ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:08.075006 3555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe9b76ad18d478c83fc930f819979e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:08.075009 3415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe9b76ad18d478c83fc930f819979e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:08.075129 3831 ts_tablet_manager.cc:1918] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:08.076220 3788 tablet_bootstrap.cc:492] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: Bootstrap starting.
I20260923 22:17:08.076311 3785 tablet_bootstrap.cc:492] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: Bootstrap starting.
I20260923 22:17:08.076401 3831 log.cc:1196] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/e62e66d0fb3542639d0e81c2533d462e
I20260923 22:17:08.076514 3788 tablet_bootstrap.cc:654] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:08.076699 3785 tablet_bootstrap.cc:654] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:08.076869 3831 ts_tablet_manager.cc:1939] T e62e66d0fb3542639d0e81c2533d462e P 0155cc3683184257ba80c730dc2d87fc: Deleting consensus metadata
I20260923 22:17:08.077246 3262 catalog_manager.cc:4841] TS 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859): tablet e62e66d0fb3542639d0e81c2533d462e (table test-table [id=e70ebaa44ef344b98d02bb100069f491]) successfully deleted
I20260923 22:17:08.077347 3785 tablet_bootstrap.cc:492] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: No bootstrap required, opened a new log
I20260923 22:17:08.077389 3785 ts_tablet_manager.cc:1397] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:08.077577 3785 raft_consensus.cc:348] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.077641 3785 raft_consensus.cc:374] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:08.077661 3785 raft_consensus.cc:729] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8d9536a7f3449a84f49f5094c0bcec, State: Initialized, Role: FOLLOWER
I20260923 22:17:08.077700 3785 consensus_queue.cc:260] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.077772 3785 ts_tablet_manager.cc:1428] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:08.078143 3784 tablet_bootstrap.cc:492] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: Bootstrap starting.
I20260923 22:17:08.078521 3784 tablet_bootstrap.cc:654] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:08.079042 3788 tablet_bootstrap.cc:492] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: No bootstrap required, opened a new log
I20260923 22:17:08.079090 3788 ts_tablet_manager.cc:1397] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 22:17:08.079190 3788 raft_consensus.cc:348] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.079249 3788 raft_consensus.cc:374] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:08.079273 3788 raft_consensus.cc:729] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e2b107e43649c8b9cf87ef6f6e859d, State: Initialized, Role: FOLLOWER
I20260923 22:17:08.079314 3788 consensus_queue.cc:260] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.079396 3788 ts_tablet_manager.cc:1428] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:08.079499 3784 tablet_bootstrap.cc:492] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: No bootstrap required, opened a new log
I20260923 22:17:08.079546 3784 ts_tablet_manager.cc:1397] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:08.079670 3784 raft_consensus.cc:348] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.079726 3784 raft_consensus.cc:374] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:08.079746 3784 raft_consensus.cc:729] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0155cc3683184257ba80c730dc2d87fc, State: Initialized, Role: FOLLOWER
I20260923 22:17:08.079802 3784 consensus_queue.cc:260] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.079895 3784 ts_tablet_manager.cc:1428] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:08.086760 3833 tablet_replica.cc:330] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: stopping tablet replica
I20260923 22:17:08.086889 3833 raft_consensus.cc:2229] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:08.086949 3833 raft_consensus.cc:2258] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:08.087065 3832 tablet_replica.cc:330] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: stopping tablet replica
I20260923 22:17:08.087214 3832 raft_consensus.cc:2229] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:08.087354 3832 raft_consensus.cc:2258] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:08.087373 3833 ts_tablet_manager.cc:1905] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:08.087862 3832 ts_tablet_manager.cc:1905] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:08.089382 3833 ts_tablet_manager.cc:1918] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:08.089455 3833 log.cc:1196] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/e62e66d0fb3542639d0e81c2533d462e
I20260923 22:17:08.089722 3833 ts_tablet_manager.cc:1939] T e62e66d0fb3542639d0e81c2533d462e P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting consensus metadata
I20260923 22:17:08.089908 3832 ts_tablet_manager.cc:1918] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:08.090022 3832 log.cc:1196] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/e62e66d0fb3542639d0e81c2533d462e
I20260923 22:17:08.090127 3264 catalog_manager.cc:4841] TS b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113): tablet e62e66d0fb3542639d0e81c2533d462e (table test-table [id=e70ebaa44ef344b98d02bb100069f491]) successfully deleted
I20260923 22:17:08.090382 3832 ts_tablet_manager.cc:1939] T e62e66d0fb3542639d0e81c2533d462e P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting consensus metadata
I20260923 22:17:08.090767 3264 catalog_manager.cc:4841] TS 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059): tablet e62e66d0fb3542639d0e81c2533d462e (table test-table [id=e70ebaa44ef344b98d02bb100069f491]) successfully deleted
I20260923 22:17:08.210393 3794 raft_consensus.cc:482] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:08.210489 3794 raft_consensus.cc:504] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.210704 3794 leader_election.cc:288] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:08.210948 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe9b76ad18d478c83fc930f819979e9" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" is_pre_election: true
I20260923 22:17:08.210937 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe9b76ad18d478c83fc930f819979e9" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc" is_pre_election: true
I20260923 22:17:08.211030 3575 raft_consensus.cc:2454] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 0.
I20260923 22:17:08.211055 3715 raft_consensus.cc:2454] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 0.
I20260923 22:17:08.211191 3368 leader_election.cc:302] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8d9536a7f3449a84f49f5094c0bcec, b8e2b107e43649c8b9cf87ef6f6e859d; no voters:
I20260923 22:17:08.211283 3794 raft_consensus.cc:2781] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:08.211337 3794 raft_consensus.cc:482] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:08.211362 3794 raft_consensus.cc:3037] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:08.212070 3794 raft_consensus.cc:504] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.212205 3794 leader_election.cc:288] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 election: Requested vote from peers b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:08.212357 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe9b76ad18d478c83fc930f819979e9" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d"
I20260923 22:17:08.212357 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe9b76ad18d478c83fc930f819979e9" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc"
I20260923 22:17:08.212430 3575 raft_consensus.cc:3037] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:08.212445 3715 raft_consensus.cc:3037] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:08.213017 3575 raft_consensus.cc:2454] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 1.
I20260923 22:17:08.213040 3715 raft_consensus.cc:2454] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 1.
I20260923 22:17:08.213205 3366 leader_election.cc:302] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0155cc3683184257ba80c730dc2d87fc, 6d8d9536a7f3449a84f49f5094c0bcec; no voters:
I20260923 22:17:08.213315 3794 raft_consensus.cc:2781] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:08.213397 3794 raft_consensus.cc:686] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 LEADER]: Becoming Leader. State: Replica: 6d8d9536a7f3449a84f49f5094c0bcec, State: Running, Role: LEADER
I20260923 22:17:08.213450 3794 consensus_queue.cc:237] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:08.214037 3277 catalog_manager.cc:5495] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65). New cstate: current_term: 1 leader_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:08.229463 3715 raft_consensus.cc:1261] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Refusing update from remote peer 6d8d9536a7f3449a84f49f5094c0bcec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:08.229485 3575 raft_consensus.cc:1261] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Refusing update from remote peer 6d8d9536a7f3449a84f49f5094c0bcec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:08.229700 3794 consensus_queue.cc:1035] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:08.229826 3826 consensus_queue.cc:1035] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:08.231326 3811 mvcc.cc:205] Tried to move back new op lower bound from 7332666688426500096 to 7332666688362651648. Current Snapshot: MvccSnapshot[applied={T|T < 7332666688426500096}]
W20260923 22:17:08.234563 3814 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 22:17:09.751689 3278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:09.751794 3278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:09.752609 3278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: Sending DeleteTablet for 3 replicas of tablet ffe9b76ad18d478c83fc930f819979e9
I20260923 22:17:09.752863 3690 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe9b76ad18d478c83fc930f819979e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:09 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:54156
I20260923 22:17:09.752890 3415 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe9b76ad18d478c83fc930f819979e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:09 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:46112
I20260923 22:17:09.752885 3555 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe9b76ad18d478c83fc930f819979e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:09 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:37358
I20260923 22:17:09.753098 3863 tablet_replica.cc:330] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: stopping tablet replica
I20260923 22:17:09.753111 3862 tablet_replica.cc:330] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: stopping tablet replica
I20260923 22:17:09.753165 3862 raft_consensus.cc:2229] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:09.753170 3863 raft_consensus.cc:2229] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:09.753204 3863 raft_consensus.cc:2258] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:09.753223 3862 raft_consensus.cc:2258] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:09.753487 3862 ts_tablet_manager.cc:1905] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:09.753486 3863 ts_tablet_manager.cc:1905] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:09.755182 3861 tablet_replica.cc:330] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: stopping tablet replica
I20260923 22:17:09.755314 3861 raft_consensus.cc:2229] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:09.755362 3861 raft_consensus.cc:2258] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:09.755508 3862 ts_tablet_manager.cc:1918] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:09.755508 3863 ts_tablet_manager.cc:1918] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:09.755566 3862 log.cc:1196] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/ffe9b76ad18d478c83fc930f819979e9
I20260923 22:17:09.755657 3863 log.cc:1196] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/ffe9b76ad18d478c83fc930f819979e9
I20260923 22:17:09.755812 3862 ts_tablet_manager.cc:1939] T ffe9b76ad18d478c83fc930f819979e9 P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting consensus metadata
I20260923 22:17:09.755882 3863 ts_tablet_manager.cc:1939] T ffe9b76ad18d478c83fc930f819979e9 P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting consensus metadata
I20260923 22:17:09.756029 3861 ts_tablet_manager.cc:1905] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:09.756170 3264 catalog_manager.cc:4841] TS b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113): tablet ffe9b76ad18d478c83fc930f819979e9 (table test-table [id=072e11c4988446cd9ab5491b545c31ec]) successfully deleted
I20260923 22:17:09.756251 3264 catalog_manager.cc:4841] TS 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059): tablet ffe9b76ad18d478c83fc930f819979e9 (table test-table [id=072e11c4988446cd9ab5491b545c31ec]) successfully deleted
I20260923 22:17:09.757603 3861 ts_tablet_manager.cc:1918] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:09.757666 3861 log.cc:1196] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/ffe9b76ad18d478c83fc930f819979e9
I20260923 22:17:09.757869 3861 ts_tablet_manager.cc:1939] T ffe9b76ad18d478c83fc930f819979e9 P 0155cc3683184257ba80c730dc2d87fc: Deleting consensus metadata
I20260923 22:17:09.758193 3262 catalog_manager.cc:4841] TS 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859): tablet ffe9b76ad18d478c83fc930f819979e9 (table test-table [id=072e11c4988446cd9ab5491b545c31ec]) successfully deleted
Sep 23 22:17:09 dist-test-slave-dmw7 krb5kdc[3233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201829, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:09 dist-test-slave-dmw7 krb5kdc[3233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201829, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:09.776587 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:09.769604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.1.129.126:34521)
0923 22:17:09.769725 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:09.769727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:09.769737 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:09.769822 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:09.769822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:09.769834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:09.769932 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:09.770522 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:09.771124 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:09.771913 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:09.772195 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:09.772269 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:09.774366 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:09.774368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:09.774368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:09.774375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:09.775690 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:09.775986 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:09.775986 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:09.775987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:09.776035 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:09.776192 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:09.776192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:09.776193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:09.776298 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:09.776372 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:09.776458 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 22:17:09.776517 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260923 22:17:09.778014 3278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:09.778196 3278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:09.779579 3555 tablet_service.cc:1467] Processing CreateTablet for tablet 431f7fbf068d471b911f9ba2eeea87ef (DEFAULT_TABLE table=test-table [id=19c9f61ce7e045c79e46a64cd0e7359c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:09.779590 3690 tablet_service.cc:1467] Processing CreateTablet for tablet 431f7fbf068d471b911f9ba2eeea87ef (DEFAULT_TABLE table=test-table [id=19c9f61ce7e045c79e46a64cd0e7359c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:09.779639 3415 tablet_service.cc:1467] Processing CreateTablet for tablet 431f7fbf068d471b911f9ba2eeea87ef (DEFAULT_TABLE table=test-table [id=19c9f61ce7e045c79e46a64cd0e7359c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:09.779718 3690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f7fbf068d471b911f9ba2eeea87ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:09.779718 3555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f7fbf068d471b911f9ba2eeea87ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:09.779740 3415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f7fbf068d471b911f9ba2eeea87ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:09.781050 3872 tablet_bootstrap.cc:492] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: Bootstrap starting.
I20260923 22:17:09.781229 3873 tablet_bootstrap.cc:492] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: Bootstrap starting.
I20260923 22:17:09.781416 3872 tablet_bootstrap.cc:654] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:09.781620 3873 tablet_bootstrap.cc:654] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:09.781966 3874 tablet_bootstrap.cc:492] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: Bootstrap starting.
I20260923 22:17:09.782058 3872 tablet_bootstrap.cc:492] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: No bootstrap required, opened a new log
I20260923 22:17:09.782109 3872 ts_tablet_manager.cc:1397] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:09.782259 3873 tablet_bootstrap.cc:492] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: No bootstrap required, opened a new log
I20260923 22:17:09.782231 3872 raft_consensus.cc:348] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.782311 3872 raft_consensus.cc:374] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:09.782325 3873 ts_tablet_manager.cc:1397] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:09.782327 3874 tablet_bootstrap.cc:654] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:09.782343 3872 raft_consensus.cc:729] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e2b107e43649c8b9cf87ef6f6e859d, State: Initialized, Role: FOLLOWER
I20260923 22:17:09.782414 3872 consensus_queue.cc:260] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.782454 3873 raft_consensus.cc:348] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.782526 3873 raft_consensus.cc:374] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:09.782545 3873 raft_consensus.cc:729] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0155cc3683184257ba80c730dc2d87fc, State: Initialized, Role: FOLLOWER
I20260923 22:17:09.782603 3873 consensus_queue.cc:260] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.782680 3872 ts_tablet_manager.cc:1428] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:09.782805 3873 ts_tablet_manager.cc:1428] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:09.783103 3874 tablet_bootstrap.cc:492] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: No bootstrap required, opened a new log
I20260923 22:17:09.783159 3874 ts_tablet_manager.cc:1397] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:09.783283 3874 raft_consensus.cc:348] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.783350 3874 raft_consensus.cc:374] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:09.783365 3874 raft_consensus.cc:729] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8d9536a7f3449a84f49f5094c0bcec, State: Initialized, Role: FOLLOWER
I20260923 22:17:09.783421 3874 consensus_queue.cc:260] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.783519 3874 ts_tablet_manager.cc:1428] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:09.910996 3856 raft_consensus.cc:482] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:09.911101 3856 raft_consensus.cc:504] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.911310 3856 leader_election.cc:288] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:09.911559 3435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f7fbf068d471b911f9ba2eeea87ef" candidate_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" is_pre_election: true
I20260923 22:17:09.911588 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f7fbf068d471b911f9ba2eeea87ef" candidate_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc" is_pre_election: true
I20260923 22:17:09.911664 3435 raft_consensus.cc:2454] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e2b107e43649c8b9cf87ef6f6e859d in term 0.
I20260923 22:17:09.911665 3715 raft_consensus.cc:2454] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e2b107e43649c8b9cf87ef6f6e859d in term 0.
I20260923 22:17:09.911846 3506 leader_election.cc:302] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0155cc3683184257ba80c730dc2d87fc, b8e2b107e43649c8b9cf87ef6f6e859d; no voters:
I20260923 22:17:09.911973 3856 raft_consensus.cc:2781] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:09.912016 3856 raft_consensus.cc:482] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:09.912042 3856 raft_consensus.cc:3037] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:09.912708 3856 raft_consensus.cc:504] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.912846 3856 leader_election.cc:288] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [CANDIDATE]: Term 1 election: Requested vote from peers 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:09.913012 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f7fbf068d471b911f9ba2eeea87ef" candidate_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc"
I20260923 22:17:09.913012 3435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f7fbf068d471b911f9ba2eeea87ef" candidate_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8d9536a7f3449a84f49f5094c0bcec"
I20260923 22:17:09.913110 3715 raft_consensus.cc:3037] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:09.913136 3435 raft_consensus.cc:3037] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:09.913908 3435 raft_consensus.cc:2454] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e2b107e43649c8b9cf87ef6f6e859d in term 1.
I20260923 22:17:09.913908 3715 raft_consensus.cc:2454] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e2b107e43649c8b9cf87ef6f6e859d in term 1.
I20260923 22:17:09.914158 3508 leader_election.cc:302] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0155cc3683184257ba80c730dc2d87fc, 6d8d9536a7f3449a84f49f5094c0bcec, b8e2b107e43649c8b9cf87ef6f6e859d; no voters:
I20260923 22:17:09.914281 3856 raft_consensus.cc:2781] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:09.914351 3856 raft_consensus.cc:686] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 LEADER]: Becoming Leader. State: Replica: b8e2b107e43649c8b9cf87ef6f6e859d, State: Running, Role: LEADER
I20260923 22:17:09.914422 3856 consensus_queue.cc:237] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.915115 3278 catalog_manager.cc:5495] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d reported cstate change: term changed from 0 to 1, leader changed from <none> to b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66). New cstate: current_term: 1 leader_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:09.931187 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:09.928051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.1.129.66:42113)
0923 22:17:09.928228 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:09.928230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:09.928238 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:09.928266 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:09.928267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:09.928276 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:09.928551 (+ 275us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:09.928588 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:09.929192 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:09.930497 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:09.930786 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:09.930843 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:09.930970 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:09.930997 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:09.931032 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:09.931066 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 22:17:09.931084 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":116,"threads_started":1}
I20260923 22:17:09.935333 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:09.933337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:56329 (local address 127.1.129.66:42113)
0923 22:17:09.933386 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:09.933388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:09.933495 (+ 107us) server_negotiation.cc:407] Connection header received
0923 22:17:09.933533 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:09.933533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:09.933541 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:09.933582 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:09.933654 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:09.934085 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:09.934817 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:09.934985 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:09.935036 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:09.935134 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:09.935154 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:09.935186 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:09.935232 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 22:17:09.935246 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:17:09.935343 3884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:09.933297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:42113 (local address 127.1.129.66:56329)
0923 22:17:09.933429 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:09.933439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:09.933494 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:09.933587 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:09.933588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:09.933589 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:09.933621 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:09.933622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:09.934143 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:09.934144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:09.934747 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:09.934749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:09.935047 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:09.935047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:09.935083 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:09.935087 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:09.935198 (+ 111us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:09.935199 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:09.935232 (+ 33us) client_negotiation.cc:240] Negotiation successful
0923 22:17:09.935263 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":90,"threads_started":1}
I20260923 22:17:09.937238 3435 raft_consensus.cc:1261] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Refusing update from remote peer b8e2b107e43649c8b9cf87ef6f6e859d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:09.937248 3715 raft_consensus.cc:1261] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Refusing update from remote peer b8e2b107e43649c8b9cf87ef6f6e859d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:09.937457 3856 consensus_queue.cc:1035] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:09.937572 3819 consensus_queue.cc:1035] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:09.947726 3277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:09.947832 3277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:09.948527 3277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: Sending DeleteTablet for 3 replicas of tablet 431f7fbf068d471b911f9ba2eeea87ef
I20260923 22:17:09.948805 3415 tablet_service.cc:1514] Processing DeleteTablet for tablet 431f7fbf068d471b911f9ba2eeea87ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:09 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:46112
I20260923 22:17:09.948858 3690 tablet_service.cc:1514] Processing DeleteTablet for tablet 431f7fbf068d471b911f9ba2eeea87ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:09 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:54156
I20260923 22:17:09.948902 3862 tablet_replica.cc:330] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: stopping tablet replica
I20260923 22:17:09.948954 3862 raft_consensus.cc:2229] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:09.948951 3861 tablet_replica.cc:330] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: stopping tablet replica
I20260923 22:17:09.948987 3862 raft_consensus.cc:2258] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:09.949003 3861 raft_consensus.cc:2229] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:09.949043 3861 raft_consensus.cc:2258] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:09.949249 3555 tablet_service.cc:1514] Processing DeleteTablet for tablet 431f7fbf068d471b911f9ba2eeea87ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:09 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:37358
I20260923 22:17:09.949267 3277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 22:17:09.949348 3863 tablet_replica.cc:330] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: stopping tablet replica
I20260923 22:17:09.949398 3863 raft_consensus.cc:2229] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 LEADER]: Raft consensus shutting down.
W20260923 22:17:09.949410 3277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:09.949481 3863 raft_consensus.cc:2258] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:09.949793 3862 ts_tablet_manager.cc:1905] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:09.949811 3861 ts_tablet_manager.cc:1905] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:09.951197 3861 ts_tablet_manager.cc:1918] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:09.951241 3861 log.cc:1196] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/431f7fbf068d471b911f9ba2eeea87ef
I20260923 22:17:09.951320 3862 ts_tablet_manager.cc:1918] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:09.951370 3862 log.cc:1196] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/431f7fbf068d471b911f9ba2eeea87ef
I20260923 22:17:09.951412 3861 ts_tablet_manager.cc:1939] T 431f7fbf068d471b911f9ba2eeea87ef P 0155cc3683184257ba80c730dc2d87fc: Deleting consensus metadata
I20260923 22:17:09.951582 3862 ts_tablet_manager.cc:1939] T 431f7fbf068d471b911f9ba2eeea87ef P 6d8d9536a7f3449a84f49f5094c0bcec: Deleting consensus metadata
I20260923 22:17:09.951706 3262 catalog_manager.cc:4841] TS 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859): tablet 431f7fbf068d471b911f9ba2eeea87ef (table test-table [id=19c9f61ce7e045c79e46a64cd0e7359c]) successfully deleted
I20260923 22:17:09.951881 3863 ts_tablet_manager.cc:1905] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:09.952219 3264 catalog_manager.cc:4841] TS 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059): tablet 431f7fbf068d471b911f9ba2eeea87ef (table test-table [id=19c9f61ce7e045c79e46a64cd0e7359c]) successfully deleted
I20260923 22:17:09.952837 3415 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab0aebab4ea44518b8789623471a40d (DEFAULT_TABLE table=test-table [id=c6b1d8f61b954f63902aeae942485efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:09.952893 3690 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab0aebab4ea44518b8789623471a40d (DEFAULT_TABLE table=test-table [id=c6b1d8f61b954f63902aeae942485efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:09.952936 3415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab0aebab4ea44518b8789623471a40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:09.952986 3690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab0aebab4ea44518b8789623471a40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:09.953475 3863 ts_tablet_manager.cc:1918] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:09.953531 3863 log.cc:1196] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/431f7fbf068d471b911f9ba2eeea87ef
I20260923 22:17:09.953773 3863 ts_tablet_manager.cc:1939] T 431f7fbf068d471b911f9ba2eeea87ef P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting consensus metadata
I20260923 22:17:09.954061 3874 tablet_bootstrap.cc:492] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec: Bootstrap starting.
I20260923 22:17:09.954267 3873 tablet_bootstrap.cc:492] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: Bootstrap starting.
I20260923 22:17:09.954324 3264 catalog_manager.cc:4841] TS b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113): tablet 431f7fbf068d471b911f9ba2eeea87ef (table test-table [id=19c9f61ce7e045c79e46a64cd0e7359c]) successfully deleted
I20260923 22:17:09.954368 3874 tablet_bootstrap.cc:654] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:09.954612 3873 tablet_bootstrap.cc:654] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:09.954710 3555 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab0aebab4ea44518b8789623471a40d (DEFAULT_TABLE table=test-table [id=c6b1d8f61b954f63902aeae942485efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:09.954833 3555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab0aebab4ea44518b8789623471a40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:09.954988 3874 tablet_bootstrap.cc:492] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec: No bootstrap required, opened a new log
I20260923 22:17:09.955041 3874 ts_tablet_manager.cc:1397] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:09.955137 3874 raft_consensus.cc:348] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.955201 3874 raft_consensus.cc:374] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:09.955229 3874 raft_consensus.cc:729] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8d9536a7f3449a84f49f5094c0bcec, State: Initialized, Role: FOLLOWER
I20260923 22:17:09.955268 3874 consensus_queue.cc:260] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.955381 3874 ts_tablet_manager.cc:1428] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:09.955957 3872 tablet_bootstrap.cc:492] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: Bootstrap starting.
I20260923 22:17:09.956343 3872 tablet_bootstrap.cc:654] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:09.956914 3872 tablet_bootstrap.cc:492] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: No bootstrap required, opened a new log
I20260923 22:17:09.956982 3872 ts_tablet_manager.cc:1397] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:09.957087 3872 raft_consensus.cc:348] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.957176 3872 raft_consensus.cc:374] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:09.957212 3872 raft_consensus.cc:729] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e2b107e43649c8b9cf87ef6f6e859d, State: Initialized, Role: FOLLOWER
I20260923 22:17:09.957264 3872 consensus_queue.cc:260] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.957356 3872 ts_tablet_manager.cc:1428] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:09.965431 3873 tablet_bootstrap.cc:492] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: No bootstrap required, opened a new log
I20260923 22:17:09.965487 3873 ts_tablet_manager.cc:1397] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260923 22:17:09.965637 3873 raft_consensus.cc:348] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.965691 3873 raft_consensus.cc:374] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:09.965705 3873 raft_consensus.cc:729] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0155cc3683184257ba80c730dc2d87fc, State: Initialized, Role: FOLLOWER
I20260923 22:17:09.965755 3873 consensus_queue.cc:260] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:09.965874 3873 ts_tablet_manager.cc:1428] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:10.123080 3826 raft_consensus.cc:482] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:10.123193 3826 raft_consensus.cc:504] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:10.123428 3826 leader_election.cc:288] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:10.123692 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab0aebab4ea44518b8789623471a40d" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc" is_pre_election: true
I20260923 22:17:10.123706 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab0aebab4ea44518b8789623471a40d" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" is_pre_election: true
I20260923 22:17:10.123786 3715 raft_consensus.cc:2454] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 0.
I20260923 22:17:10.123821 3575 raft_consensus.cc:2454] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 0.
I20260923 22:17:10.123958 3366 leader_election.cc:302] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0155cc3683184257ba80c730dc2d87fc, 6d8d9536a7f3449a84f49f5094c0bcec; no voters:
I20260923 22:17:10.124073 3826 raft_consensus.cc:2781] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:10.124130 3826 raft_consensus.cc:482] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:10.124145 3826 raft_consensus.cc:3037] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:10.124819 3826 raft_consensus.cc:504] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:10.124958 3826 leader_election.cc:288] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 election: Requested vote from peers b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113), 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859)
I20260923 22:17:10.125121 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab0aebab4ea44518b8789623471a40d" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d"
I20260923 22:17:10.125197 3575 raft_consensus.cc:3037] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:10.125170 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab0aebab4ea44518b8789623471a40d" candidate_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0155cc3683184257ba80c730dc2d87fc"
I20260923 22:17:10.125249 3715 raft_consensus.cc:3037] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:10.125792 3575 raft_consensus.cc:2454] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 1.
I20260923 22:17:10.125797 3715 raft_consensus.cc:2454] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8d9536a7f3449a84f49f5094c0bcec in term 1.
I20260923 22:17:10.125958 3368 leader_election.cc:302] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8d9536a7f3449a84f49f5094c0bcec, b8e2b107e43649c8b9cf87ef6f6e859d; no voters:
I20260923 22:17:10.126104 3826 raft_consensus.cc:2781] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:10.126164 3826 raft_consensus.cc:686] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [term 1 LEADER]: Becoming Leader. State: Replica: 6d8d9536a7f3449a84f49f5094c0bcec, State: Running, Role: LEADER
I20260923 22:17:10.126202 3826 consensus_queue.cc:237] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } }
I20260923 22:17:10.126849 3278 catalog_manager.cc:5495] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65). New cstate: current_term: 1 leader_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8d9536a7f3449a84f49f5094c0bcec" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 37059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:10.142647 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:10.140274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.1.129.65:37059)
0923 22:17:10.140438 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:10.140440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:10.140447 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:10.140474 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:10.140474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:10.140483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:10.140558 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:10.140663 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:10.141147 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:10.141885 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:10.142099 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:10.142183 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:10.142342 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:10.142375 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:10.142445 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:10.142505 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 22:17:10.142525 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260923 22:17:10.144654 3715 raft_consensus.cc:1261] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Refusing update from remote peer 6d8d9536a7f3449a84f49f5094c0bcec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:10.144661 3575 raft_consensus.cc:1261] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Refusing update from remote peer 6d8d9536a7f3449a84f49f5094c0bcec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:10.144866 3854 consensus_queue.cc:1035] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0155cc3683184257ba80c730dc2d87fc" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:10.144979 3826 consensus_queue.cc:1035] T 1ab0aebab4ea44518b8789623471a40d P 6d8d9536a7f3449a84f49f5094c0bcec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e2b107e43649c8b9cf87ef6f6e859d" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:17:10.149284 3844 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 22:17:11.667145 3277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:11.667265 3277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:11.668072 3277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c17139bc6a34e549474f2f9d17ac779: Sending DeleteTablet for 3 replicas of tablet 1ab0aebab4ea44518b8789623471a40d
I20260923 22:17:11.668318 3415 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab0aebab4ea44518b8789623471a40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:11 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:46112
I20260923 22:17:11.668313 3690 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab0aebab4ea44518b8789623471a40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:11 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:54156
I20260923 22:17:11.668464 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 3350
I20260923 22:17:11.668545 3555 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab0aebab4ea44518b8789623471a40d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:11 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:37358
I20260923 22:17:11.669330 3920 tablet_replica.cc:330] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: stopping tablet replica
I20260923 22:17:11.669410 3920 raft_consensus.cc:2229] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:11.669454 3920 raft_consensus.cc:2258] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:11.669721 3920 ts_tablet_manager.cc:1905] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:11.671824 3920 ts_tablet_manager.cc:1918] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:11.671906 3920 log.cc:1196] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/1ab0aebab4ea44518b8789623471a40d
I20260923 22:17:11.672130 3920 ts_tablet_manager.cc:1939] T 1ab0aebab4ea44518b8789623471a40d P b8e2b107e43649c8b9cf87ef6f6e859d: Deleting consensus metadata
I20260923 22:17:11.672171 3918 tablet_replica.cc:330] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: stopping tablet replica
I20260923 22:17:11.672233 3918 raft_consensus.cc:2229] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:11.672286 3918 raft_consensus.cc:2258] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:11.672562 3264 catalog_manager.cc:4841] TS b8e2b107e43649c8b9cf87ef6f6e859d (127.1.129.66:42113): tablet 1ab0aebab4ea44518b8789623471a40d (table test-table [id=c6b1d8f61b954f63902aeae942485efa]) successfully deleted
I20260923 22:17:11.672680 3918 ts_tablet_manager.cc:1905] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:11.674039 3918 ts_tablet_manager.cc:1918] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:11.674093 3918 log.cc:1196] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/1ab0aebab4ea44518b8789623471a40d
I20260923 22:17:11.674278 3918 ts_tablet_manager.cc:1939] T 1ab0aebab4ea44518b8789623471a40d P 0155cc3683184257ba80c730dc2d87fc: Deleting consensus metadata
I20260923 22:17:11.674654 3262 catalog_manager.cc:4841] TS 0155cc3683184257ba80c730dc2d87fc (127.1.129.67:34859): tablet 1ab0aebab4ea44518b8789623471a40d (table test-table [id=c6b1d8f61b954f63902aeae942485efa]) successfully deleted
W20260923 22:17:11.676649 3264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:37059 (error 108)
I20260923 22:17:11.676885 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 3490
I20260923 22:17:11.677728 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.677018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:37059 (local address 127.0.0.1:46134)
0923 22:17:11.677627 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:11.677676 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":57,"threads_started":1}
W20260923 22:17:11.677824 3264 catalog_manager.cc:4584] TS 6d8d9536a7f3449a84f49f5094c0bcec (127.1.129.65:37059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ab0aebab4ea44518b8789623471a40d: Network error: Client connection negotiation failed: client connection to 127.1.129.65:37059: connect: Connection refused (error 111)
I20260923 22:17:11.683326 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 3630
I20260923 22:17:11.688081 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 3246
2026-09-23T22:17:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5810 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:11 dist-test-slave-dmw7 krb5kdc[3925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:11 dist-test-slave-dmw7 krb5kdc[3925](info): set up 2 sockets
Sep 23 22:17:11 dist-test-slave-dmw7 krb5kdc[3925](info): commencing operation
krb5kdc: starting...
W20260923 22:17:12.742396 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:12 dist-test-slave-dmw7 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:12.857107 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36661
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:12.935503 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:12.935739 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:12.935776 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:12.936936 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:12.937007 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:12.937031 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:12.937052 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:12.937103 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:12.938990 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:12.939258 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:12.939517 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:12.944876 3938 server_base.cc:1034] running on GCE node
W20260923 22:17:12.944890 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:12.944896 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:12.944895 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:12.945356 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:12.945641 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:12.946825 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790201832946819 us; error 37 us; skew 500 ppm
Sep 23 22:17:12 dist-test-slave-dmw7 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201832, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:12.952512 3938 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:12.952934 3938 webserver.cc:460] Webserver started at http://127.1.129.126:40673/ using document root <none> and password file <none>
I20260923 22:17:12.953120 3938 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:12.953159 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:12.953244 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:12.954200 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "717ccdee335545c69f5af59c2415204b"
format_stamp: "Formatted at 2026-09-23 22:17:12 on dist-test-slave-dmw7"
server_key: "edd5cce8db346b3b74e0b55a8f7b8873"
server_key_iv: "9b1e5a5e4e8a8719c91c8fde7ae00054"
server_key_version: "encryptionkey@0"
I20260923 22:17:12.954499 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "717ccdee335545c69f5af59c2415204b"
format_stamp: "Formatted at 2026-09-23 22:17:12 on dist-test-slave-dmw7"
server_key: "edd5cce8db346b3b74e0b55a8f7b8873"
server_key_iv: "9b1e5a5e4e8a8719c91c8fde7ae00054"
server_key_version: "encryptionkey@0"
I20260923 22:17:12.955782 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:12.956419 3953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:12.956611 3938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:12.956700 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "717ccdee335545c69f5af59c2415204b"
format_stamp: "Formatted at 2026-09-23 22:17:12 on dist-test-slave-dmw7"
server_key: "edd5cce8db346b3b74e0b55a8f7b8873"
server_key_iv: "9b1e5a5e4e8a8719c91c8fde7ae00054"
server_key_version: "encryptionkey@0"
I20260923 22:17:12.956753 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:12.968747 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:12.971186 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:12.971314 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:12.975224 3938 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:36661
I20260923 22:17:12.975360 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:36661 every 8 connection(s)
I20260923 22:17:12.975643 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:12.976521 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:12.978837 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b: Bootstrap starting.
I20260923 22:17:12.979444 4006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:12.979746 4006 log.cc:824] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:12.980372 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b: No bootstrap required, opened a new log
I20260923 22:17:12.981606 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 3938
I20260923 22:17:12.981675 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:12.981838 1541 external_mini_cluster.cc:1428] Setting key c7ffe6c2f11e41115eca9f70a551a259
I20260923 22:17:12.982070 4006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717ccdee335545c69f5af59c2415204b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36661 } }
I20260923 22:17:12.982250 4006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:12.982295 4006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717ccdee335545c69f5af59c2415204b, State: Initialized, Role: FOLLOWER
I20260923 22:17:12.982417 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717ccdee335545c69f5af59c2415204b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36661 } }
I20260923 22:17:12.982506 4006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:12.982551 4006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:12.982609 4006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:12.983654 4006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717ccdee335545c69f5af59c2415204b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36661 } }
I20260923 22:17:12.983762 4006 leader_election.cc:302] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 717ccdee335545c69f5af59c2415204b; no voters:
I20260923 22:17:12.983906 4006 leader_election.cc:288] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:12.983979 4011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:12.984115 4011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [term 1 LEADER]: Becoming Leader. State: Replica: 717ccdee335545c69f5af59c2415204b, State: Running, Role: LEADER
I20260923 22:17:12.984148 4006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:12.984200 4011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717ccdee335545c69f5af59c2415204b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36661 } }
I20260923 22:17:12.984508 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 717ccdee335545c69f5af59c2415204b. Latest consensus state: current_term: 1 leader_uuid: "717ccdee335545c69f5af59c2415204b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717ccdee335545c69f5af59c2415204b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36661 } } }
I20260923 22:17:12.984570 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:12.984615 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "717ccdee335545c69f5af59c2415204b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717ccdee335545c69f5af59c2415204b" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36661 } } }
I20260923 22:17:12.984685 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:12.984776 4015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:12.985286 4015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:12.987337 4015 catalog_manager.cc:1347] Generated new cluster ID: 4bf4ffe2119d4dceb2f00717c50a8707
I20260923 22:17:12.987388 4015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:12 dist-test-slave-dmw7 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:12.993482 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:12.982347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.1.129.126:36661)
0923 22:17:12.982592 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:12.982601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:12.982655 (+ 54us) server_negotiation.cc:407] Connection header received
0923 22:17:12.982782 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:12.982790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:12.982844 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:12.983019 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:12.983572 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:12.985053 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:12.986020 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:12.986620 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:12.986757 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:12.989425 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:12.989429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:12.989430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:12.989442 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:12.991454 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:12.991856 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:12.991860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:12.991860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:12.991983 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:12.992192 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:12.992193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:12.992194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:12.992497 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:12.993049 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:12.993133 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 22:17:12.993275 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260923 22:17:12.994472 4015 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:12.995165 4015 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:13.012195 4015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 717ccdee335545c69f5af59c2415204b: Generated new TSK 0
I20260923 22:17:13.012429 4015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:13.039433 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36661
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:13.118956 4032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:13.119167 4032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:13.119202 4032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:13.120299 4032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:13.120360 4032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:13.120404 4032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:13.121780 4032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:13.122026 4032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:13.122236 4032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:13.126978 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.126958 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.127005 4040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:13.127270 4032 server_base.cc:1034] running on GCE node
I20260923 22:17:13.127481 4032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:13.127656 4032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:13.128821 4032 hybrid_clock.cc:648] HybridClock initialized: now 1790201833128815 us; error 42 us; skew 500 ppm
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:13.131302 4032 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:13.131754 4032 webserver.cc:460] Webserver started at http://127.1.129.65:46039/ using document root <none> and password file <none>
I20260923 22:17:13.131970 4032 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:13.132035 4032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:13.132186 4032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:13.133277 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "9ffc59d249bb414e816bede8f67c9010"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "11f589ba8ec83570230eb11a97804e8d"
server_key_iv: "5c2bf20062afb943cbfc94efaf63f4f0"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.133641 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "9ffc59d249bb414e816bede8f67c9010"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "11f589ba8ec83570230eb11a97804e8d"
server_key_iv: "5c2bf20062afb943cbfc94efaf63f4f0"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.134944 4032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:13.135648 4047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.135811 4032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:13.135902 4032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "9ffc59d249bb414e816bede8f67c9010"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "11f589ba8ec83570230eb11a97804e8d"
server_key_iv: "5c2bf20062afb943cbfc94efaf63f4f0"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.135975 4032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:13.144100 4032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:13.146239 4032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:13.146394 4032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:13.146596 4032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:13.146929 4032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:13.146994 4032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.147033 4032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:13.147060 4032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.152765 4032 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39819
I20260923 22:17:13.152819 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39819 every 8 connection(s)
I20260923 22:17:13.153098 4032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:13.154219 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 4032
I20260923 22:17:13.154297 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:13.154479 1541 external_mini_cluster.cc:1428] Setting key 3bdfa390a4e21f5a09249b30bdaa64a7
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
I20260923 22:17:13.163143 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.153818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:41947 (local address 127.1.129.126:36661)
0923 22:17:13.153872 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.153874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.154092 (+ 218us) server_negotiation.cc:407] Connection header received
0923 22:17:13.154721 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.154722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.154737 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.154784 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.155709 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.156341 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.157294 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.157653 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.157725 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.160092 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.160094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.160094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.160106 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.161738 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.162121 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.162122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.162123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.162186 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.162425 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.162426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.162426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.162571 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.162686 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.162937 (+ 251us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.163042 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:17:13.163309 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.153799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36661 (local address 127.1.129.65:41947)
0923 22:17:13.153961 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.153993 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.154592 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.154848 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.154852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.155411 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:13.155585 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.155589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.156439 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.156440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.157215 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.157217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.157737 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.157738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.157844 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.158361 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:13.158371 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:13.160006 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:13.161829 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.161834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.161836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.162045 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.162255 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.162256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.162256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.162354 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.162679 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:13.162685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:13.162802 (+ 117us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.162887 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.163042 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260923 22:17:13.163959 4161 heartbeater.cc:347] Connected to a master server at 127.1.129.126:36661
I20260923 22:17:13.164040 4161 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:13.164177 4161 heartbeater.cc:511] Master 127.1.129.126:36661 requested a full tablet report, sending...
I20260923 22:17:13.164656 3970 ts_manager.cc:194] Registered new tserver with Master: 9ffc59d249bb414e816bede8f67c9010 (127.1.129.65:39819)
I20260923 22:17:13.165221 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:41947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:13.181701 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36661
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:13.261960 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:13.262179 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:13.262203 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:13.263410 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:13.263451 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:13.263478 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:13.264861 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:13.265094 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:13.265326 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:13.271993 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:13.272006 4166 server_base.cc:1034] running on GCE node
W20260923 22:17:13.272279 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.272456 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:13.272696 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:13.272961 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:13.274122 4166 hybrid_clock.cc:648] HybridClock initialized: now 1790201833274112 us; error 42 us; skew 500 ppm
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:13.276612 4166 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:13.277086 4166 webserver.cc:460] Webserver started at http://127.1.129.66:43225/ using document root <none> and password file <none>
I20260923 22:17:13.277319 4166 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:13.277386 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:13.277539 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:13.278595 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "20267c16c2ba4e33b0cb1bf405c35168"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "03522cb05f39c1108a0cf57d3db569f2"
server_key_iv: "d6a57aef922c7cc4fbd21970c14845fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.279017 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "20267c16c2ba4e33b0cb1bf405c35168"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "03522cb05f39c1108a0cf57d3db569f2"
server_key_iv: "d6a57aef922c7cc4fbd21970c14845fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.280330 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:13.281028 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.281200 4166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:13.281302 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "20267c16c2ba4e33b0cb1bf405c35168"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "03522cb05f39c1108a0cf57d3db569f2"
server_key_iv: "d6a57aef922c7cc4fbd21970c14845fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.281386 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:13.305481 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:13.307699 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:13.307894 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:13.308143 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:13.308475 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:13.308530 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.308573 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:13.308614 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.314383 4166 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:37101
I20260923 22:17:13.314432 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:37101 every 8 connection(s)
I20260923 22:17:13.314788 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:13.316766 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 4166
I20260923 22:17:13.316867 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:13.317054 1541 external_mini_cluster.cc:1428] Setting key 2978069a7513eb3aa026df57179f43d8
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
I20260923 22:17:13.324910 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.315515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:40017 (local address 127.1.129.126:36661)
0923 22:17:13.315562 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.315564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.315784 (+ 220us) server_negotiation.cc:407] Connection header received
0923 22:17:13.316234 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.316235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.316246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.316297 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.317518 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.318101 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.319569 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.319785 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.319875 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.322031 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.322034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.322034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.322042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.323522 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.323943 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.323945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.323945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.324010 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.324255 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.324255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.324256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.324380 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.324449 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.324728 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.324820 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:13.325090 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.315503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36661 (local address 127.1.129.66:40017)
0923 22:17:13.315667 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.315703 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.316156 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.316356 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.316359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.317170 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:13.317433 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.317439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.318192 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.318193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.319477 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.319480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.319885 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.319886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.320015 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.320534 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:13.320548 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:13.321934 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:13.323633 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.323640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.323642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.323867 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.324075 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.324076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.324077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.324184 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.324452 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:13.324459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:13.324594 (+ 135us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.324694 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.324856 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260923 22:17:13.325609 4295 heartbeater.cc:347] Connected to a master server at 127.1.129.126:36661
I20260923 22:17:13.325706 4295 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:13.325894 4295 heartbeater.cc:511] Master 127.1.129.126:36661 requested a full tablet report, sending...
I20260923 22:17:13.326297 3970 ts_manager.cc:194] Registered new tserver with Master: 20267c16c2ba4e33b0cb1bf405c35168 (127.1.129.66:37101)
I20260923 22:17:13.326711 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:40017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:13.341816 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36661
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:13.421748 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:13.421952 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:13.421975 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:13.423192 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:13.423243 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:13.423269 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:13.424710 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:13.424961 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:13.425196 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:13.430080 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.430092 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.430090 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:13.430253 4300 server_base.cc:1034] running on GCE node
I20260923 22:17:13.430456 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:13.430720 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:13.431860 4300 hybrid_clock.cc:648] HybridClock initialized: now 1790201833431847 us; error 28 us; skew 500 ppm
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:13.434280 4300 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:13.434765 4300 webserver.cc:460] Webserver started at http://127.1.129.67:34593/ using document root <none> and password file <none>
I20260923 22:17:13.435006 4300 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:13.435072 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:13.435220 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:13.436254 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "85ce398b620549578d979671d873e92c"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "76b5ec4b1cb1d6014b148f80103d03a2"
server_key_iv: "1c595381343be569feac750b9ac993fb"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.436619 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "85ce398b620549578d979671d873e92c"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "76b5ec4b1cb1d6014b148f80103d03a2"
server_key_iv: "1c595381343be569feac750b9ac993fb"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.437894 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:13.438848 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.439110 4300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:13.439241 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "85ce398b620549578d979671d873e92c"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-dmw7"
server_key: "76b5ec4b1cb1d6014b148f80103d03a2"
server_key_iv: "1c595381343be569feac750b9ac993fb"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.439330 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:13.462669 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:13.465060 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:13.465251 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:13.465499 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:13.465821 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:13.465874 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.465915 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:13.465940 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.471585 4300 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44619
I20260923 22:17:13.471625 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44619 every 8 connection(s)
I20260923 22:17:13.471958 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:13.476657 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 4300
I20260923 22:17:13.476758 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:13.476946 1541 external_mini_cluster.cc:1428] Setting key 5c9fc661369bfc2b613ea5aa3a172988
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:13.480971 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.472696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:39067 (local address 127.1.129.126:36661)
0923 22:17:13.472772 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.472774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.472946 (+ 172us) server_negotiation.cc:407] Connection header received
0923 22:17:13.473389 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.473389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.473399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.473450 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.474309 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.474867 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.475793 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.475969 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.476017 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.478347 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.478348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.478348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.478354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.479778 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.480114 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.480115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.480115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.480203 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.480415 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.480416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.480416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.480520 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.480571 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.480841 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.480910 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 22:17:13.481041 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.472665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36661 (local address 127.1.129.67:39067)
0923 22:17:13.472830 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.472865 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.473311 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.473504 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.473507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.474049 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:13.474235 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.474240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.474939 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.474940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.475712 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.475713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.476036 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.476037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.476144 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.476577 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:13.476587 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:13.478272 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:13.479842 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.479847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.479848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.480049 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.480269 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.480270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.480270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.480359 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.480575 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:13.480579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:13.480686 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.480773 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.480887 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260923 22:17:13.481567 4429 heartbeater.cc:347] Connected to a master server at 127.1.129.126:36661
I20260923 22:17:13.481665 4429 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:13.481851 4429 heartbeater.cc:511] Master 127.1.129.126:36661 requested a full tablet report, sending...
I20260923 22:17:13.482211 3970 ts_manager.cc:194] Registered new tserver with Master: 85ce398b620549578d979671d873e92c (127.1.129.67:44619)
I20260923 22:17:13.482573 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:39067
I20260923 22:17:13.488571 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 22:17:13.499521 4010 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:13.498493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.1.129.126:36661)
0923 22:17:13.498581 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.498583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.498697 (+ 114us) server_negotiation.cc:407] Connection header received
0923 22:17:13.498819 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.498820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.498831 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.498898 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.499433 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35668: BlockingRecv error: recv got EOF from 127.0.0.1:35668 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 22:17:13.500442 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 4032
I20260923 22:17:13.504971 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 4166
I20260923 22:17:13.509140 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 4300
I20260923 22:17:13.513227 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 3938
2026-09-23T22:17:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1823 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[4441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[4441](info): set up 2 sockets
Sep 23 22:17:13 dist-test-slave-dmw7 krb5kdc[4441](info): commencing operation
krb5kdc: starting...
W20260923 22:17:14.550400 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:14 dist-test-slave-dmw7 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:14.665021 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36817
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:14.745883 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:14.746065 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:14.746085 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:14.747257 4454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:14.747295 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:14.747309 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:14.747361 4454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:14.747396 4454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:14.748904 4454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:14.749140 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:14.749399 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:14.754570 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:14.754725 4454 server_base.cc:1034] running on GCE node
W20260923 22:17:14.754577 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:14.754601 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:14.755039 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:14.755345 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:14.756513 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790201834756495 us; error 30 us; skew 500 ppm
Sep 23 22:17:14 dist-test-slave-dmw7 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:14.758869 4454 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:14.759240 4454 webserver.cc:460] Webserver started at http://127.1.129.126:45707/ using document root <none> and password file <none>
I20260923 22:17:14.759482 4454 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:14.759527 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:14.759667 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:14.760658 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "74313b8d3e664d45844808d960f45e90"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-dmw7"
server_key: "7febd868654e541ffe1781e7bdf58bcb"
server_key_iv: "5a1b658d3a046f50936f8bb410000b12"
server_key_version: "encryptionkey@0"
I20260923 22:17:14.760998 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "74313b8d3e664d45844808d960f45e90"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-dmw7"
server_key: "7febd868654e541ffe1781e7bdf58bcb"
server_key_iv: "5a1b658d3a046f50936f8bb410000b12"
server_key_version: "encryptionkey@0"
I20260923 22:17:14.762262 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:14.763031 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:14.763199 4454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:14.763309 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "74313b8d3e664d45844808d960f45e90"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-dmw7"
server_key: "7febd868654e541ffe1781e7bdf58bcb"
server_key_iv: "5a1b658d3a046f50936f8bb410000b12"
server_key_version: "encryptionkey@0"
I20260923 22:17:14.763389 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:14.791441 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:14.793891 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:14.794075 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:14.797986 4454 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:36817
I20260923 22:17:14.798020 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:36817 every 8 connection(s)
I20260923 22:17:14.798372 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:14.799244 4522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:14.800072 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 4454
I20260923 22:17:14.800146 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:14.800331 1541 external_mini_cluster.cc:1428] Setting key 55c1f2424f647e35d43dabcd97dfa1e1
I20260923 22:17:14.802424 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90: Bootstrap starting.
I20260923 22:17:14.803083 4522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:14.803371 4522 log.cc:824] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:14.804112 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90: No bootstrap required, opened a new log
I20260923 22:17:14.806571 4522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313b8d3e664d45844808d960f45e90" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36817 } }
I20260923 22:17:14.806798 4522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:14.806849 4522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74313b8d3e664d45844808d960f45e90, State: Initialized, Role: FOLLOWER
I20260923 22:17:14.806973 4522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313b8d3e664d45844808d960f45e90" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36817 } }
I20260923 22:17:14.807068 4522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:14.807111 4522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:14.807163 4522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 22:17:14 dist-test-slave-dmw7 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:14.807855 4522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313b8d3e664d45844808d960f45e90" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36817 } }
I20260923 22:17:14.807976 4522 leader_election.cc:302] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74313b8d3e664d45844808d960f45e90; no voters:
I20260923 22:17:14.808130 4522 leader_election.cc:288] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:14.808292 4527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:14.808413 4522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:14.808473 4527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [term 1 LEADER]: Becoming Leader. State: Replica: 74313b8d3e664d45844808d960f45e90, State: Running, Role: LEADER
I20260923 22:17:14.809129 4527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313b8d3e664d45844808d960f45e90" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36817 } }
I20260923 22:17:14.809687 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74313b8d3e664d45844808d960f45e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313b8d3e664d45844808d960f45e90" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36817 } } }
I20260923 22:17:14.809767 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:14.809975 4542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:14.810189 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74313b8d3e664d45844808d960f45e90. Latest consensus state: current_term: 1 leader_uuid: "74313b8d3e664d45844808d960f45e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313b8d3e664d45844808d960f45e90" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36817 } } }
I20260923 22:17:14.810487 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:14.810911 4542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:14.811691 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:14.800878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.1.129.126:36817)
0923 22:17:14.801053 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:14.801060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:14.801085 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:17:14.801495 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:14.801501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:14.801548 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:14.801656 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:14.802457 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:14.803470 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:14.804443 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:14.804833 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:14.804914 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:14.807757 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:14.807760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:14.807761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:14.807770 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:14.809901 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:14.810337 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:14.810341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:14.810341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:14.810434 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:14.810751 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:14.810752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:14.810753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:14.811052 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:14.811216 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:14.811350 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 22:17:14.811484 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260923 22:17:14.812487 4542 catalog_manager.cc:1347] Generated new cluster ID: ff6a1ffcecd54d45bb85dbedc82a4ed0
I20260923 22:17:14.812537 4542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:14.812525 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:14.824419 4542 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:14.825136 4542 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:14.829061 4542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74313b8d3e664d45844808d960f45e90: Generated new TSK 0
I20260923 22:17:14.829252 4542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:14.832683 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:14.828529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.1.129.126:36817)
0923 22:17:14.828572 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:14.828574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:14.828813 (+ 239us) server_negotiation.cc:407] Connection header received
0923 22:17:14.828839 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:14.828840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:14.828850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:14.828909 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:14.829398 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:14.830131 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:14.831243 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:14.831453 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:14.831511 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:14.832202 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:14.832204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:14.832204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:14.832389 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:14.832405 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:14.832411 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:14.832458 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:14.832541 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 22:17:14.832606 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:14.834939 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 4454
2026-09-23T22:17:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1319 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:14 dist-test-slave-dmw7 krb5kdc[4555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:14 dist-test-slave-dmw7 krb5kdc[4555](info): set up 2 sockets
Sep 23 22:17:14 dist-test-slave-dmw7 krb5kdc[4555](info): commencing operation
krb5kdc: starting...
W20260923 22:17:15.855563 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:15 dist-test-slave-dmw7 krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:15.973439 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:46489
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:16.053282 4568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:16.053527 4568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:16.053563 4568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:16.054774 4568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:16.054847 4568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:16.054869 4568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:16.054890 4568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:16.054920 4568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:16.056424 4568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:16.056710 4568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:16.056936 4568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:16.061941 4573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:16.061920 4574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:16.061920 4576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:16.061954 4568 server_base.cc:1034] running on GCE node
I20260923 22:17:16.062412 4568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:16.062773 4568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:16.063916 4568 hybrid_clock.cc:648] HybridClock initialized: now 1790201836063903 us; error 30 us; skew 500 ppm
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:16.066406 4568 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:16.066906 4568 webserver.cc:460] Webserver started at http://127.1.129.126:34047/ using document root <none> and password file <none>
I20260923 22:17:16.067145 4568 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:16.067193 4568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:16.067370 4568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:16.068440 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "e50d933d96286f3c8b3a9bdd1af5dbe2"
server_key_iv: "31425d29d369f737edda541534e40b2f"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.068799 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "e50d933d96286f3c8b3a9bdd1af5dbe2"
server_key_iv: "31425d29d369f737edda541534e40b2f"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.070170 4568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:16.071017 4583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.071197 4568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.071305 4568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "e50d933d96286f3c8b3a9bdd1af5dbe2"
server_key_iv: "31425d29d369f737edda541534e40b2f"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.071395 4568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:16.082183 4568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:16.084731 4568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:16.084911 4568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:16.089092 4568 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:46489
I20260923 22:17:16.089128 4635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:46489 every 8 connection(s)
I20260923 22:17:16.089493 4568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:16.090541 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:16.093500 4636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1: Bootstrap starting.
I20260923 22:17:16.094084 4636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:16.094379 4636 log.cc:824] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:16.095155 4636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1: No bootstrap required, opened a new log
I20260923 22:17:16.096725 4636 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46489 } }
I20260923 22:17:16.096864 4636 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:16.096889 4636 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ff5b44e8fe4a249fe6f59f6e752ae1, State: Initialized, Role: FOLLOWER
I20260923 22:17:16.097055 4636 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46489 } }
I20260923 22:17:16.097116 4636 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:16.097138 4636 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:16.097167 4636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:16.097759 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 4568
I20260923 22:17:16.097882 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:16.098075 1541 external_mini_cluster.cc:1428] Setting key cf27b917bc024516a110b1f730dff1c8
I20260923 22:17:16.098093 4636 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46489 } }
I20260923 22:17:16.098209 4636 leader_election.cc:302] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ff5b44e8fe4a249fe6f59f6e752ae1; no voters:
I20260923 22:17:16.098415 4636 leader_election.cc:288] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:16.098438 4639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:16.098804 4639 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [term 1 LEADER]: Becoming Leader. State: Replica: b4ff5b44e8fe4a249fe6f59f6e752ae1, State: Running, Role: LEADER
I20260923 22:17:16.098953 4639 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46489 } }
I20260923 22:17:16.098798 4636 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:16.099305 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46489 } } }
I20260923 22:17:16.099320 4643 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ff5b44e8fe4a249fe6f59f6e752ae1. Latest consensus state: current_term: 1 leader_uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ff5b44e8fe4a249fe6f59f6e752ae1" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46489 } } }
I20260923 22:17:16.099465 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:16.099539 4643 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:16.099972 4645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:16.100451 4645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:16.102491 4645 catalog_manager.cc:1347] Generated new cluster ID: b0ebb16266254bf98c46c1567c5ce999
I20260923 22:17:16.102541 4645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:16.108944 4645 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:16.109619 4645 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:16.110405 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.098558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.1.129.126:46489)
0923 22:17:16.100728 (+ 2170us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.100738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.100776 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:17:16.101159 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.101168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.101229 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.101399 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:16.102155 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.103231 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.104031 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.104468 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.104595 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.107000 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:16.107004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:16.107005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:16.107022 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:16.108774 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.109193 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.109196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.109197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.109285 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.109500 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.109501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.109501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.109767 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:16.109977 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.110045 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.110195 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2115,"thread_start_us":97,"threads_started":1}
I20260923 22:17:16.112790 4645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ff5b44e8fe4a249fe6f59f6e752ae1: Generated new TSK 0
I20260923 22:17:16.112943 4645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:16.141393 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46489
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:16.232427 4662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:16.232630 4662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:16.232653 4662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:16.233785 4662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:16.233824 4662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:16.233855 4662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:16.235345 4662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:16.235590 4662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:16.235823 4662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:16.240715 4667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:16.240720 4668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:16.240975 4662 server_base.cc:1034] running on GCE node
W20260923 22:17:16.240734 4670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:16.241288 4662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:16.241535 4662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:16.242693 4662 hybrid_clock.cc:648] HybridClock initialized: now 1790201836242677 us; error 30 us; skew 500 ppm
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:16.245388 4662 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:16.245870 4662 webserver.cc:460] Webserver started at http://127.1.129.65:37661/ using document root <none> and password file <none>
I20260923 22:17:16.246122 4662 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:16.246176 4662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:16.246299 4662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:16.247610 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "5809ed72b55d405093ccffbffee3ca73"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "4ff129ea1916375e98e7b1e18cc531ff"
server_key_iv: "4416fc097176a8378e8505f9dc63a4b9"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.247987 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "5809ed72b55d405093ccffbffee3ca73"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "4ff129ea1916375e98e7b1e18cc531ff"
server_key_iv: "4416fc097176a8378e8505f9dc63a4b9"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.249666 4662 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 22:17:16.250500 4677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.250720 4662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:16.250821 4662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "5809ed72b55d405093ccffbffee3ca73"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "4ff129ea1916375e98e7b1e18cc531ff"
server_key_iv: "4416fc097176a8378e8505f9dc63a4b9"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.250891 4662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:16.273044 4662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:16.275610 4662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:16.275729 4662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:16.275926 4662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:16.276262 4662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:16.276300 4662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.276324 4662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:16.276371 4662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.281970 4662 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:40897
I20260923 22:17:16.282037 4790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:40897 every 8 connection(s)
I20260923 22:17:16.282372 4662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:16.286792 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 4662
I20260923 22:17:16.286903 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:16.287102 1541 external_mini_cluster.cc:1428] Setting key 65db03c0333c1d74b2cd9bcba6ef1bd5
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:16.291620 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.283143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:38901 (local address 127.1.129.126:46489)
0923 22:17:16.283212 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.283214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.283439 (+ 225us) server_negotiation.cc:407] Connection header received
0923 22:17:16.283902 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.283903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.283912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.283971 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:16.284898 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.285386 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.286329 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.286516 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.286563 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.288819 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:16.288821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:16.288821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:16.288828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:16.290226 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.290659 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.290660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.290660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.290724 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.290978 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.290979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.290980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.291095 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:16.291172 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.291438 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.291531 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:16.291817 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.283112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46489 (local address 127.1.129.65:38901)
0923 22:17:16.283296 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:16.283345 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:16.283822 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:16.284027 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:16.284031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:16.284628 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:16.284820 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.284825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.285463 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.285463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.286255 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.286256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.286615 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.286616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.286795 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.287375 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:16.287383 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:16.288709 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:16.290335 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:16.290342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:16.290344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:16.290566 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:16.290796 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:16.290797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:16.290797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:16.290911 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:16.291160 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:16.291166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:16.291308 (+ 142us) client_negotiation.cc:769] Sending connection context
0923 22:17:16.291416 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 22:17:16.291572 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260923 22:17:16.292341 4791 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46489
I20260923 22:17:16.292451 4791 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:16.292689 4791 heartbeater.cc:511] Master 127.1.129.126:46489 requested a full tablet report, sending...
I20260923 22:17:16.293263 4600 ts_manager.cc:194] Registered new tserver with Master: 5809ed72b55d405093ccffbffee3ca73 (127.1.129.65:40897)
I20260923 22:17:16.294241 4600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:38901
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:16.313257 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46489
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:16.393862 4796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:16.394053 4796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:16.394076 4796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:16.395165 4796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:16.395198 4796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:16.395222 4796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:16.396557 4796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:16.396733 4796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:16.396899 4796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:16.401929 4804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:16.401971 4802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:16.402097 4796 server_base.cc:1034] running on GCE node
W20260923 22:17:16.401976 4801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:16.402391 4796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:16.402686 4796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:16.403841 4796 hybrid_clock.cc:648] HybridClock initialized: now 1790201836403824 us; error 30 us; skew 500 ppm
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:16.406212 4796 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:16.406723 4796 webserver.cc:460] Webserver started at http://127.1.129.66:39003/ using document root <none> and password file <none>
I20260923 22:17:16.406972 4796 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:16.407040 4796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:16.407186 4796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:16.408252 4796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "26c4ab58307d42449a1c7cba77f8c2fa"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "9132f506b84bf4f6fe3748affd4cfa6f"
server_key_iv: "a262bf5981cf36ca75b94b98cb4a65b3"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.408615 4796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "26c4ab58307d42449a1c7cba77f8c2fa"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "9132f506b84bf4f6fe3748affd4cfa6f"
server_key_iv: "a262bf5981cf36ca75b94b98cb4a65b3"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.409909 4796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:16.410686 4811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.410872 4796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:16.410970 4796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "26c4ab58307d42449a1c7cba77f8c2fa"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "9132f506b84bf4f6fe3748affd4cfa6f"
server_key_iv: "a262bf5981cf36ca75b94b98cb4a65b3"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.411023 4796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:16.434415 4796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:16.436395 4796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:16.436508 4796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:16.436686 4796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:16.437005 4796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:16.437040 4796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.437062 4796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:16.437151 4796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.442855 4796 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:35413
I20260923 22:17:16.442935 4924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:35413 every 8 connection(s)
I20260923 22:17:16.443243 4796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:16.448047 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 4796
I20260923 22:17:16.448159 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:16.448379 1541 external_mini_cluster.cc:1428] Setting key bb18df2c9261dedcd41d6285d766d045
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:16.452746 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.443976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:37627 (local address 127.1.129.126:46489)
0923 22:17:16.444035 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.444037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.444238 (+ 201us) server_negotiation.cc:407] Connection header received
0923 22:17:16.444722 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.444724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.444735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.444798 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:16.445759 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.446336 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.447341 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.447533 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.447588 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.449886 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:16.449887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:16.449887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:16.449894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:16.451359 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.451773 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.451775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.451775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.451845 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.452086 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.452087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.452087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.452238 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:16.452298 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.452577 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.452654 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:17:16.452951 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.443934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46489 (local address 127.1.129.66:37627)
0923 22:17:16.444084 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:16.444125 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:16.444614 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:16.444854 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:16.444857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:16.445473 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:16.445663 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.445668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.446419 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.446420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.447257 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.447260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.447606 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.447607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.447704 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.448169 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:16.448180 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:16.449764 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:16.451459 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:16.451466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:16.451468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:16.451685 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:16.451912 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:16.451913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:16.451913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:16.452018 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:16.452304 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:16.452310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:16.452448 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 22:17:16.452556 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 22:17:16.452712 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260923 22:17:16.453588 4925 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46489
I20260923 22:17:16.453704 4925 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:16.453908 4925 heartbeater.cc:511] Master 127.1.129.126:46489 requested a full tablet report, sending...
I20260923 22:17:16.454300 4600 ts_manager.cc:194] Registered new tserver with Master: 26c4ab58307d42449a1c7cba77f8c2fa (127.1.129.66:35413)
I20260923 22:17:16.454749 4600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:37627
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:16.473814 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46489
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:16.554364 4930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:16.554620 4930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:16.554659 4930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:16.555780 4930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:16.555815 4930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:16.555841 4930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:16.557225 4930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:16.557406 4930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:16.557572 4930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:16.562430 4935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:16.562476 4936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:16.562482 4938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:16.562633 4930 server_base.cc:1034] running on GCE node
I20260923 22:17:16.562943 4930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:16.563225 4930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:16.564411 4930 hybrid_clock.cc:648] HybridClock initialized: now 1790201836564396 us; error 25 us; skew 500 ppm
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:16.566912 4930 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:16.567443 4930 webserver.cc:460] Webserver started at http://127.1.129.67:38077/ using document root <none> and password file <none>
I20260923 22:17:16.567691 4930 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:16.567761 4930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:16.567899 4930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:16.568930 4930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "3c4df476b8cc4e9c831dfa159c1d8e11"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "50e637c7a8cbc3ede4a75ce1ade34b38"
server_key_iv: "b0158bd535abd0be36888139b144aebc"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.569307 4930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "3c4df476b8cc4e9c831dfa159c1d8e11"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "50e637c7a8cbc3ede4a75ce1ade34b38"
server_key_iv: "b0158bd535abd0be36888139b144aebc"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.570604 4930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:16.571414 4945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.571583 4930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:16.571694 4930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "3c4df476b8cc4e9c831dfa159c1d8e11"
format_stamp: "Formatted at 2026-09-23 22:17:16 on dist-test-slave-dmw7"
server_key: "50e637c7a8cbc3ede4a75ce1ade34b38"
server_key_iv: "b0158bd535abd0be36888139b144aebc"
server_key_version: "encryptionkey@0"
I20260923 22:17:16.571764 4930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:16.593528 4930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:16.595856 4930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:16.596033 4930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:16.596247 4930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:16.596573 4930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:16.596613 4930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.596634 4930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:16.596647 4930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:16.602541 4930 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44161
I20260923 22:17:16.602617 5058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44161 every 8 connection(s)
I20260923 22:17:16.602949 4930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:16.609016 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 4930
I20260923 22:17:16.609098 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:16.609313 1541 external_mini_cluster.cc:1428] Setting key 7acc1ded82e1e9c7ce8d76cb87c96112
I20260923 22:17:16.612033 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.603625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:44747 (local address 127.1.129.126:46489)
0923 22:17:16.603676 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.603679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.603861 (+ 182us) server_negotiation.cc:407] Connection header received
0923 22:17:16.604392 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.604393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.604405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.604460 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:16.605353 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.605867 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.606994 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.607179 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.607271 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.609363 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:16.609366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:16.609366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:16.609394 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:16.610957 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.611302 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.611303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.611303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.611355 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.611555 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.611555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.611556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.611648 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:16.611697 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.611919 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.611974 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:17:16.612188 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.603600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46489 (local address 127.1.129.67:44747)
0923 22:17:16.603730 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:16.603780 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:16.604280 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:16.604529 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:16.604533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:16.605072 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:16.605259 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.605264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.605969 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.605970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.606887 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.606889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.607276 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.607277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.607398 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.607794 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:16.607802 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:16.609256 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:16.611025 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:16.611030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:16.611031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:16.611236 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:16.611413 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:16.611413 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:16.611413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:16.611502 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:16.611700 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:16.611703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:16.611805 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 22:17:16.611892 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:17:16.612000 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260923 22:17:16.612632 5059 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46489
I20260923 22:17:16.612752 5059 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:16.612926 5059 heartbeater.cc:511] Master 127.1.129.126:46489 requested a full tablet report, sending...
I20260923 22:17:16.613243 4600 ts_manager.cc:194] Registered new tserver with Master: 3c4df476b8cc4e9c831dfa159c1d8e11 (127.1.129.67:44161)
I20260923 22:17:16.613695 4600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:44747
I20260923 22:17:16.621140 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201836, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:16.643364 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.636371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.1.129.126:46489)
0923 22:17:16.636442 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.636444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.636598 (+ 154us) server_negotiation.cc:407] Connection header received
0923 22:17:16.636629 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.636630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.636641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.636687 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:16.637250 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.637830 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.638528 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.638767 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.638821 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.640935 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:16.640936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:16.640936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:16.640943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:16.642337 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.642620 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.642621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.642621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.642724 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.642888 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.642888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.642888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.643020 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:16.643088 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.643190 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.643247 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260923 22:17:16.643636 4600 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45718
I20260923 22:17:16.644102 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 4662
I20260923 22:17:16.648468 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 4796
I20260923 22:17:16.652875 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 4930
I20260923 22:17:16.656900 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 4568
2026-09-23T22:17:16Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1824 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[5071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[5071](info): set up 2 sockets
Sep 23 22:17:16 dist-test-slave-dmw7 krb5kdc[5071](info): commencing operation
krb5kdc: starting...
W20260923 22:17:17.690568 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:17 dist-test-slave-dmw7 krb5kdc[5071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:17.810112 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34149
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:34149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:17.894385 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:17.894692 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:17.894738 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:17.895951 5084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:17.896040 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:17.896068 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:17.896095 5084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:17.896179 5084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:17.897799 5084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:34149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:17.898120 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:17.898375 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:17.903386 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:17.903398 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:17.903434 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:17.903501 5084 server_base.cc:1034] running on GCE node
I20260923 22:17:17.903869 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:17.904182 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:17.905340 5084 hybrid_clock.cc:648] HybridClock initialized: now 1790201837905324 us; error 26 us; skew 500 ppm
Sep 23 22:17:17 dist-test-slave-dmw7 krb5kdc[5071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:17.908066 5084 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:17.908584 5084 webserver.cc:460] Webserver started at http://127.1.129.126:42953/ using document root <none> and password file <none>
I20260923 22:17:17.908831 5084 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:17.908880 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:17.909010 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:17.910091 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "a5b3cde6a9f440fda70169df9fffc2fa"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-dmw7"
server_key: "364d890d0b6e7bd4b177afd8d32a47c6"
server_key_iv: "63bcceffd2660d0490d8ef3b5ddf7a43"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.910449 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "a5b3cde6a9f440fda70169df9fffc2fa"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-dmw7"
server_key: "364d890d0b6e7bd4b177afd8d32a47c6"
server_key_iv: "63bcceffd2660d0490d8ef3b5ddf7a43"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.911940 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:17.912889 5099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.913326 5084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:17.913442 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "a5b3cde6a9f440fda70169df9fffc2fa"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-dmw7"
server_key: "364d890d0b6e7bd4b177afd8d32a47c6"
server_key_iv: "63bcceffd2660d0490d8ef3b5ddf7a43"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.913522 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:17.924113 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:17.926120 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:17.926268 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:17.930588 5084 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:34149
I20260923 22:17:17.930624 5151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:34149 every 8 connection(s)
I20260923 22:17:17.931275 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:17.931989 5152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:17.934335 5152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa: Bootstrap starting.
I20260923 22:17:17.934862 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 5084
I20260923 22:17:17.934942 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:17.935087 5152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:17.935138 1541 external_mini_cluster.cc:1428] Setting key 1c67a327214451fe9b5d85f2f9006dec
I20260923 22:17:17.935398 5152 log.cc:824] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:17.936210 5152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa: No bootstrap required, opened a new log
I20260923 22:17:17.938688 5152 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b3cde6a9f440fda70169df9fffc2fa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34149 } }
I20260923 22:17:17.938871 5152 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:17.938913 5152 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b3cde6a9f440fda70169df9fffc2fa, State: Initialized, Role: FOLLOWER
I20260923 22:17:17.939064 5152 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b3cde6a9f440fda70169df9fffc2fa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34149 } }
I20260923 22:17:17.939174 5152 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:17.939219 5152 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:17.939268 5152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:17.939939 5152 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b3cde6a9f440fda70169df9fffc2fa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34149 } }
I20260923 22:17:17.940047 5152 leader_election.cc:302] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5b3cde6a9f440fda70169df9fffc2fa; no voters:
I20260923 22:17:17.940218 5152 leader_election.cc:288] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:17.940301 5157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:17.940470 5152 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:17.940502 5157 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [term 1 LEADER]: Becoming Leader. State: Replica: a5b3cde6a9f440fda70169df9fffc2fa, State: Running, Role: LEADER
I20260923 22:17:17.940726 5157 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b3cde6a9f440fda70169df9fffc2fa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34149 } }
I20260923 22:17:17.941075 5158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5b3cde6a9f440fda70169df9fffc2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b3cde6a9f440fda70169df9fffc2fa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34149 } } }
I20260923 22:17:17.941082 5159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5b3cde6a9f440fda70169df9fffc2fa. Latest consensus state: current_term: 1 leader_uuid: "a5b3cde6a9f440fda70169df9fffc2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b3cde6a9f440fda70169df9fffc2fa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34149 } } }
I20260923 22:17:17.941201 5158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:17.941210 5159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:17.941411 5166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:17.941947 5166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 22:17:17 dist-test-slave-dmw7 krb5kdc[5071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:17.943603 5166 catalog_manager.cc:1347] Generated new cluster ID: 32b632814e854a85a9aaaa0dd09c034c
I20260923 22:17:17.943661 5166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:17.948282 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.935674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.1.129.126:34149)
0923 22:17:17.935833 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.935842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.935900 (+ 58us) server_negotiation.cc:407] Connection header received
0923 22:17:17.936519 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.936528 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.936595 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.936759 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.937373 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.938860 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.939778 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.940454 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.940582 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.944166 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.944170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.944171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.944183 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.946186 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.946620 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.946623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.946623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.946750 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.946969 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.946969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.946970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.947331 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.947634 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.947904 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.948059 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260923 22:17:17.952206 5166 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:17.952951 5166 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:17.960296 5166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5b3cde6a9f440fda70169df9fffc2fa: Generated new TSK 0
I20260923 22:17:17.960556 5166 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:17.997217 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34149
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:18.096062 5178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:18.096289 5178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:18.096325 5178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:18.097446 5178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:18.097507 5178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:18.097553 5178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:18.099066 5178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:18.099319 5178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:18.099601 5178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:18.104668 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:18.104660 5184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:18.104656 5183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:18.104882 5178 server_base.cc:1034] running on GCE node
I20260923 22:17:18.105206 5178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:18.105388 5178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:18.106570 5178 hybrid_clock.cc:648] HybridClock initialized: now 1790201838106555 us; error 28 us; skew 500 ppm
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:18.114037 5178 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:18.114573 5178 webserver.cc:460] Webserver started at http://127.1.129.65:33137/ using document root <none> and password file <none>
I20260923 22:17:18.114843 5178 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:18.114915 5178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:18.115069 5178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:18.116094 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "dd4a9d8c59ec45c69a99ee9d1b107c73"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "0c70ec2474666c52317bfd9b70dc628f"
server_key_iv: "28552557078821f09068536364443b0e"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.116451 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "dd4a9d8c59ec45c69a99ee9d1b107c73"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "0c70ec2474666c52317bfd9b70dc628f"
server_key_iv: "28552557078821f09068536364443b0e"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.117802 5178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:18.118475 5193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.118683 5178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.118819 5178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "dd4a9d8c59ec45c69a99ee9d1b107c73"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "0c70ec2474666c52317bfd9b70dc628f"
server_key_iv: "28552557078821f09068536364443b0e"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.118898 5178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:18.130355 5178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:18.132361 5178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:18.132529 5178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:18.132761 5178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:18.133126 5178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:18.133181 5178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.133216 5178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:18.133244 5178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.139763 5178 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:42971
I20260923 22:17:18.139874 5306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:42971 every 8 connection(s)
I20260923 22:17:18.140185 5178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:18.142815 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 5178
I20260923 22:17:18.142916 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:18.143121 1541 external_mini_cluster.cc:1428] Setting key 265ac60e5e4c46781b51d7b15af648a5
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
I20260923 22:17:18.152830 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.141198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:35561 (local address 127.1.129.126:34149)
0923 22:17:18.141254 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.141256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.141299 (+ 43us) server_negotiation.cc:407] Connection header received
0923 22:17:18.142654 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.142656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.142669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.142728 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:18.143586 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.144351 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.145947 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.146179 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.146265 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.149159 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:18.149160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:18.149161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:18.149168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:18.151087 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.151627 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.151629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.151629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.151691 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.152001 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.152002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.152002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.152140 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:18.152206 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.152556 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.152675 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:17:18.153373 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.140969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34149 (local address 127.1.129.65:35561)
0923 22:17:18.141133 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.141184 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.142523 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.142843 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.142847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.143223 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:18.143490 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.143496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.144461 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.144462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.145858 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.145861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.146276 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.146277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.146410 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.147017 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:18.147027 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:18.149055 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:18.151194 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:18.151202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:18.151204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:18.151530 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:18.151770 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:18.151771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:18.151771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:18.151909 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:18.152196 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:18.152203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:18.152378 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.152923 (+ 545us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.153105 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260923 22:17:18.154033 5307 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34149
I20260923 22:17:18.154147 5307 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:18.154316 5307 heartbeater.cc:511] Master 127.1.129.126:34149 requested a full tablet report, sending...
I20260923 22:17:18.154863 5116 ts_manager.cc:194] Registered new tserver with Master: dd4a9d8c59ec45c69a99ee9d1b107c73 (127.1.129.65:42971)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
I20260923 22:17:18.155794 5116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:35561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:18.170321 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34149
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:18.248775 5312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:18.249007 5312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:18.249043 5312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:18.250310 5312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:18.250370 5312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:18.250414 5312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:18.251962 5312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:18.252233 5312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:18.252461 5312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:18.257741 5318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:18.257830 5320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:18.257755 5317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:18.257958 5312 server_base.cc:1034] running on GCE node
I20260923 22:17:18.258177 5312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:18.258437 5312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:18.259632 5312 hybrid_clock.cc:648] HybridClock initialized: now 1790201838259600 us; error 61 us; skew 500 ppm
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:18.262271 5312 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:18.262787 5312 webserver.cc:460] Webserver started at http://127.1.129.66:33537/ using document root <none> and password file <none>
I20260923 22:17:18.263024 5312 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:18.263096 5312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:18.263243 5312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:18.264312 5312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "43335d93f7d44635b8d21b865bcd68b0"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "0832970e188ff9e0a0f7aa052e8bf839"
server_key_iv: "96481e1e65b2dc90758f4604ed85db2d"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.264703 5312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "43335d93f7d44635b8d21b865bcd68b0"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "0832970e188ff9e0a0f7aa052e8bf839"
server_key_iv: "96481e1e65b2dc90758f4604ed85db2d"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.266078 5312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:18.266865 5327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.267037 5312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:18.267164 5312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "43335d93f7d44635b8d21b865bcd68b0"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "0832970e188ff9e0a0f7aa052e8bf839"
server_key_iv: "96481e1e65b2dc90758f4604ed85db2d"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.267243 5312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:18.279415 5312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:18.281849 5312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:18.282027 5312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:18.282275 5312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:18.282610 5312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:18.282718 5312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.282768 5312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:18.282799 5312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.288887 5312 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:33343
I20260923 22:17:18.288952 5440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:33343 every 8 connection(s)
I20260923 22:17:18.289295 5312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:18.294893 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 5312
I20260923 22:17:18.295034 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:18.295229 1541 external_mini_cluster.cc:1428] Setting key 2218bd2432a5d3ca8add802f04a1d213
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:18.299095 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.290094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:52501 (local address 127.1.129.126:34149)
0923 22:17:18.290187 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.290189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.290355 (+ 166us) server_negotiation.cc:407] Connection header received
0923 22:17:18.291373 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.291374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.291387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.291452 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:18.292009 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.292499 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.293464 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.293655 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.293723 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.296078 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:18.296079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:18.296080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:18.296087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:18.297592 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.298111 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.298112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.298112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.298159 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.298441 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.298441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.298441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.298533 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:18.298584 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.298968 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.299033 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 22:17:18.299399 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.290056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34149 (local address 127.1.129.66:52501)
0923 22:17:18.290239 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.290273 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.291260 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.291504 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.291507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.291745 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:18.291927 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.291931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.292581 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.292582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.293370 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.293372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.293740 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.293741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.293836 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.294317 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:18.294326 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:18.295993 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:18.297676 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:18.297682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:18.297685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:18.298030 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:18.298222 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:18.298223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:18.298223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:18.298379 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:18.298595 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:18.298601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:18.298815 (+ 214us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.298940 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.299119 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260923 22:17:18.299955 5441 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34149
I20260923 22:17:18.300067 5441 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:18.300279 5441 heartbeater.cc:511] Master 127.1.129.126:34149 requested a full tablet report, sending...
I20260923 22:17:18.300671 5116 ts_manager.cc:194] Registered new tserver with Master: 43335d93f7d44635b8d21b865bcd68b0 (127.1.129.66:33343)
I20260923 22:17:18.301208 5116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:52501
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:18.322407 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34149
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:18.403734 5446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:18.404004 5446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:18.404042 5446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:18.405252 5446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:18.405328 5446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:18.405369 5446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:18.407007 5446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:18.407285 5446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:18.407528 5446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:18.412789 5451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:18.412792 5452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:18.412847 5446 server_base.cc:1034] running on GCE node
W20260923 22:17:18.412891 5454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:18.413228 5446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:18.413524 5446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:18.414701 5446 hybrid_clock.cc:648] HybridClock initialized: now 1790201838414677 us; error 35 us; skew 500 ppm
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:18.417284 5446 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:18.417799 5446 webserver.cc:460] Webserver started at http://127.1.129.67:39469/ using document root <none> and password file <none>
I20260923 22:17:18.417996 5446 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:18.418037 5446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:18.418126 5446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:18.419350 5446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "a709d7cf181742e9a9f663abb4a45d09"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "6c46c7ca5461a0b7d6b273e57045b7f8"
server_key_iv: "c142f40d1a4d7136c99f7d74ae86c782"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.419665 5446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "a709d7cf181742e9a9f663abb4a45d09"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "6c46c7ca5461a0b7d6b273e57045b7f8"
server_key_iv: "c142f40d1a4d7136c99f7d74ae86c782"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.420928 5446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:18.421576 5461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.421787 5446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:18.421905 5446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "a709d7cf181742e9a9f663abb4a45d09"
format_stamp: "Formatted at 2026-09-23 22:17:18 on dist-test-slave-dmw7"
server_key: "6c46c7ca5461a0b7d6b273e57045b7f8"
server_key_iv: "c142f40d1a4d7136c99f7d74ae86c782"
server_key_version: "encryptionkey@0"
I20260923 22:17:18.421977 5446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:18.443753 5446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:18.446165 5446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:18.446309 5446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:18.446568 5446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:18.446928 5446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:18.446969 5446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.446992 5446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:18.447005 5446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.453156 5446 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44631
I20260923 22:17:18.453200 5574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44631 every 8 connection(s)
I20260923 22:17:18.453524 5446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:18.457479 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 5446
I20260923 22:17:18.457585 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:18.457808 1541 external_mini_cluster.cc:1428] Setting key 466cede07e4b8a9dfc9859cf5a6f9dd2
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:18.462917 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.454296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:39355 (local address 127.1.129.126:34149)
0923 22:17:18.454359 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.454361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.454568 (+ 207us) server_negotiation.cc:407] Connection header received
0923 22:17:18.455573 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.455575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.455589 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.455655 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:18.456242 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.456725 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.457723 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.458001 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.458095 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.460290 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:18.460291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:18.460292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:18.460300 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:18.461497 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.461909 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.461910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.461911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.461968 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.462230 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.462231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.462231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.462352 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:18.462435 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.462746 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.462827 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:18.463121 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.454252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34149 (local address 127.1.129.67:39355)
0923 22:17:18.454411 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.454454 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.455454 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.455711 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.455715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.455975 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:18.456167 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.456170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.456810 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.456811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.457635 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.457638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.458133 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.458134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.458287 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.458844 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:18.458851 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:18.460193 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:18.461581 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:18.461588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:18.461590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:18.461810 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:18.462043 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:18.462043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:18.462044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:18.462153 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:18.462425 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:18.462432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:18.462571 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.462722 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.462891 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":96,"threads_started":1}
I20260923 22:17:18.463548 5575 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34149
I20260923 22:17:18.463661 5575 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:18.463838 5575 heartbeater.cc:511] Master 127.1.129.126:34149 requested a full tablet report, sending...
I20260923 22:17:18.464192 5116 ts_manager.cc:194] Registered new tserver with Master: a709d7cf181742e9a9f663abb4a45d09 (127.1.129.67:44631)
I20260923 22:17:18.464566 5116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:39355
I20260923 22:17:18.469358 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:18.492298 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.484458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.1.129.65:42971)
0923 22:17:18.484718 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.484722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.484734 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:18.484759 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.484761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.484775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.484842 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:18.485474 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.486197 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.487050 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.487381 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.487440 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.489559 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:18.489566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:18.489567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:18.489573 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:18.491190 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.491525 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.491528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.491528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.491582 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.491798 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.491799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.491799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.491923 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:18.492029 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.492097 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.492156 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":154,"threads_started":1}
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:18.500524 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.493616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.1.129.126:34149)
0923 22:17:18.493686 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.493689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.493744 (+ 55us) server_negotiation.cc:407] Connection header received
0923 22:17:18.493816 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.493817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.493828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.493885 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:18.494354 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.494924 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.495611 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.495822 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.495908 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.498104 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:18.498106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:18.498106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:18.498113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:18.499413 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.499714 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.499715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.499715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.499766 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.499928 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.499929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.499929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.500059 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:18.500134 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.500284 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.500398 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 22:17:18.500785 5116 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976
I20260923 22:17:18.501168 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 5178
I20260923 22:17:18.505708 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 5312
I20260923 22:17:18.509953 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 5446
I20260923 22:17:18.514226 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 5084
2026-09-23T22:17:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1858 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5587](info): set up 2 sockets
Sep 23 22:17:18 dist-test-slave-dmw7 krb5kdc[5587](info): commencing operation
krb5kdc: starting...
W20260923 22:17:19.574549 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/test-user.keytab.
W20260923 22:17:20.084066 1541 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.449s user 0.000s sys 0.001s
Sep 23 22:17:20 dist-test-slave-dmw7 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
W20260923 22:17:21.229594 1541 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.129.126: real 1.125s user 0.000s sys 0.002s
I20260923 22:17:21.230008 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:38227
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:21.311216 5600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:21.311419 5600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:21.311440 5600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:21.312567 5600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:21.312601 5600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:21.312615 5600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:21.312628 5600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:21.312639 5600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:21.314064 5600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:21.314256 5600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:21.314433 5600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:21.319083 5608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:21.319087 5605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:21.319077 5606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:21.319100 5600 server_base.cc:1034] running on GCE node
I20260923 22:17:21.319593 5600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:21.319911 5600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:21.321087 5600 hybrid_clock.cc:648] HybridClock initialized: now 1790201841321071 us; error 28 us; skew 500 ppm
Sep 23 22:17:21 dist-test-slave-dmw7 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:21.323604 5600 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:21.323994 5600 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:21.324067 5600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:21.324219 5600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:21.325278 5600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "b7da8c39985447c38fbde1ad7b05a881"
format_stamp: "Formatted at 2026-09-23 22:17:21 on dist-test-slave-dmw7"
server_key: "2de174aca4d7b726235557b6b2d51789"
server_key_iv: "09d2d50d19da065421f304f845253ee3"
server_key_version: "encryptionkey@0"
I20260923 22:17:21.325639 5600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "b7da8c39985447c38fbde1ad7b05a881"
format_stamp: "Formatted at 2026-09-23 22:17:21 on dist-test-slave-dmw7"
server_key: "2de174aca4d7b726235557b6b2d51789"
server_key_iv: "09d2d50d19da065421f304f845253ee3"
server_key_version: "encryptionkey@0"
I20260923 22:17:21.327016 5600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:21.327770 5614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:21.327950 5600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:21.328042 5600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "b7da8c39985447c38fbde1ad7b05a881"
format_stamp: "Formatted at 2026-09-23 22:17:21 on dist-test-slave-dmw7"
server_key: "2de174aca4d7b726235557b6b2d51789"
server_key_iv: "09d2d50d19da065421f304f845253ee3"
server_key_version: "encryptionkey@0"
I20260923 22:17:21.328126 5600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:21.351886 5600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:21.353781 5600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:21.353962 5600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:21.358229 5600 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:38227
I20260923 22:17:21.358253 5666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:38227 every 8 connection(s)
I20260923 22:17:21.358547 5600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:21.359469 5667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:21.361891 5667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881: Bootstrap starting.
I20260923 22:17:21.362528 5667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:21.362862 5667 log.cc:824] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:21.363579 5667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881: No bootstrap required, opened a new log
I20260923 22:17:21.365327 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 5600
I20260923 22:17:21.365455 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:21.365654 1541 external_mini_cluster.cc:1428] Setting key 07cb5e868efd9d0c097f7d9c98ff3da3
I20260923 22:17:21.365644 5667 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7da8c39985447c38fbde1ad7b05a881" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 38227 } }
I20260923 22:17:21.365849 5667 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:21.365902 5667 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7da8c39985447c38fbde1ad7b05a881, State: Initialized, Role: FOLLOWER
I20260923 22:17:21.366065 5667 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7da8c39985447c38fbde1ad7b05a881" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 38227 } }
I20260923 22:17:21.366273 5667 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:21.366329 5667 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:21.366386 5667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 22:17:21 dist-test-slave-dmw7 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:21.375658 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:21.366148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.1.129.126:38227)
0923 22:17:21.366366 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:21.366377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:21.366415 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:17:21.366832 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:21.366838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:21.366888 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:21.367031 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:21.367677 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:21.368572 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:21.369364 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:21.369785 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:21.369922 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:21.372398 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:21.372402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:21.372403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:21.372422 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:21.374127 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:21.374498 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:21.374502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:21.374503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:21.374595 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:21.374834 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:21.374834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:21.374835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:21.375098 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:21.375197 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:21.375358 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 22:17:21.375468 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260923 22:17:21.396489 5667 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7da8c39985447c38fbde1ad7b05a881" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 38227 } }
I20260923 22:17:21.396644 5667 leader_election.cc:302] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7da8c39985447c38fbde1ad7b05a881; no voters:
I20260923 22:17:21.396930 5667 leader_election.cc:288] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:21.397001 5672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:21.397220 5667 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:21.397235 5672 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [term 1 LEADER]: Becoming Leader. State: Replica: b7da8c39985447c38fbde1ad7b05a881, State: Running, Role: LEADER
I20260923 22:17:21.397493 5672 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7da8c39985447c38fbde1ad7b05a881" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 38227 } }
I20260923 22:17:21.397909 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7da8c39985447c38fbde1ad7b05a881. Latest consensus state: current_term: 1 leader_uuid: "b7da8c39985447c38fbde1ad7b05a881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7da8c39985447c38fbde1ad7b05a881" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 38227 } } }
I20260923 22:17:21.397933 5673 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7da8c39985447c38fbde1ad7b05a881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7da8c39985447c38fbde1ad7b05a881" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 38227 } } }
I20260923 22:17:21.398118 5673 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:21.398095 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:21.398515 5681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:21.399087 5681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:21.402617 5681 catalog_manager.cc:1347] Generated new cluster ID: f1732236a3b949658adba5ed9b9b4c3d
I20260923 22:17:21.402704 5681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:21.420166 5681 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:21.420745 5681 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:21.424052 5681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881: Generated new TSK 0
I20260923 22:17:21.424261 5681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
W20260923 22:17:22.823300 1541 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.129.65: real 1.396s user 0.001s sys 0.002s
I20260923 22:17:22.823844 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:38227
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:22.905217 5693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:22.905470 5693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:22.905510 5693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:22.906771 5693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:22.906832 5693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:22.906886 5693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:22.908524 5693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:22.908803 5693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:22.909045 5693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:22.914098 5699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:22.914084 5698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:22.914108 5701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:22.914433 5693 server_base.cc:1034] running on GCE node
I20260923 22:17:22.914790 5693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:22.915045 5693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:22.916189 5693 hybrid_clock.cc:648] HybridClock initialized: now 1790201842916171 us; error 45 us; skew 500 ppm
Sep 23 22:17:22 dist-test-slave-dmw7 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201842, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:22.918766 5693 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:22.919167 5693 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:22.919219 5693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:22.919375 5693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:22.920450 5693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "c5213eccc1d9412aade978ae9427263e"
format_stamp: "Formatted at 2026-09-23 22:17:22 on dist-test-slave-dmw7"
server_key: "8ab6c3a665d591ceb172b41e0ef1596b"
server_key_iv: "25d701ffba751a0435b90522047072fb"
server_key_version: "encryptionkey@0"
I20260923 22:17:22.920814 5693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "c5213eccc1d9412aade978ae9427263e"
format_stamp: "Formatted at 2026-09-23 22:17:22 on dist-test-slave-dmw7"
server_key: "8ab6c3a665d591ceb172b41e0ef1596b"
server_key_iv: "25d701ffba751a0435b90522047072fb"
server_key_version: "encryptionkey@0"
I20260923 22:17:22.922158 5693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:22.923254 5707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:22.923429 5693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:22.923537 5693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "c5213eccc1d9412aade978ae9427263e"
format_stamp: "Formatted at 2026-09-23 22:17:22 on dist-test-slave-dmw7"
server_key: "8ab6c3a665d591ceb172b41e0ef1596b"
server_key_iv: "25d701ffba751a0435b90522047072fb"
server_key_version: "encryptionkey@0"
I20260923 22:17:22.923611 5693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:22.931878 5693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:22.933802 5693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:22.933964 5693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:22.934201 5693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:22.934512 5693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:22.934569 5693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:22.934610 5693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:22.934700 5693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:22.940634 5693 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39477
I20260923 22:17:22.940685 5820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39477 every 8 connection(s)
I20260923 22:17:22.941001 5693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:22 dist-test-slave-dmw7 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201842, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:22.948680 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 5693
I20260923 22:17:22.948797 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:22.948984 1541 external_mini_cluster.cc:1428] Setting key a09ce98c4fffbbe49b589e3424db7341
I20260923 22:17:22.950788 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:22.941805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:40557 (local address 127.1.129.126:38227)
0923 22:17:22.941950 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:22.941952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:22.942019 (+ 67us) server_negotiation.cc:407] Connection header received
0923 22:17:22.942544 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:22.942545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:22.942555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:22.942615 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:22.943589 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:22.944086 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:22.945102 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:22.945285 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:22.945334 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:22.947559 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:22.947560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:22.947560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:22.947566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:22.949137 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:22.949675 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:22.949677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:22.949677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:22.949735 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:22.950020 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:22.950021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:22.950022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:22.950176 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:22.950249 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:22.950515 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 22:17:22.950612 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260923 22:17:22.950878 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:22.941756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:38227 (local address 127.1.129.65:40557)
0923 22:17:22.941900 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:22.941953 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:22.942449 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:22.942691 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:22.942695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:22.943013 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:22.943504 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:22.943509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:22.944165 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:22.944166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:22.945025 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:22.945027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:22.945383 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:22.945384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:22.945495 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:22.946061 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:22.946069 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:22.947480 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:22.949225 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:22.949232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:22.949234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:22.949552 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:22.949805 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:22.949806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:22.949806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:22.949946 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:22.950262 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:22.950266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:22.950384 (+ 118us) client_negotiation.cc:769] Sending connection context
0923 22:17:22.950476 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 22:17:22.950605 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260923 22:17:22.951661 5821 heartbeater.cc:347] Connected to a master server at 127.1.129.126:38227
I20260923 22:17:22.951781 5821 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:22.951979 5821 heartbeater.cc:511] Master 127.1.129.126:38227 requested a full tablet report, sending...
I20260923 22:17:22.952502 5631 ts_manager.cc:194] Registered new tserver with Master: c5213eccc1d9412aade978ae9427263e (127.1.129.65:39477)
I20260923 22:17:22.953107 5631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:40557
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
I20260923 22:17:23.954068 5821 heartbeater.cc:503] Master 127.1.129.126:38227 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
W20260923 22:17:24.084564 1541 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.129.66: real 1.135s user 0.000s sys 0.003s
I20260923 22:17:24.084954 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:38227
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:24.167521 5827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:24.167730 5827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:24.167757 5827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:24.168941 5827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:24.168990 5827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:24.169019 5827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:24.170596 5827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:24.170877 5827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:24.171131 5827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:24.176142 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.176307 5827 server_base.cc:1034] running on GCE node
W20260923 22:17:24.176241 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:24.176290 5832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.176693 5827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:24.176937 5827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:24.178078 5827 hybrid_clock.cc:648] HybridClock initialized: now 1790201844178069 us; error 32 us; skew 500 ppm
Sep 23 22:17:24 dist-test-slave-dmw7 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201844, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:24.185429 5827 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:24.185873 5827 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:24.185946 5827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:24.186081 5827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:24.187201 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "e9fc578742064c0bbbde4dca6fe877a1"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "1f0f50e01f3c7c5b9162b2ccdb8cae61"
server_key_iv: "e5e30427603f8b5d3d259e55ed2e460f"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.187556 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "e9fc578742064c0bbbde4dca6fe877a1"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "1f0f50e01f3c7c5b9162b2ccdb8cae61"
server_key_iv: "e5e30427603f8b5d3d259e55ed2e460f"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.188827 5827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:24.189591 5841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.189791 5827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:24.189899 5827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "e9fc578742064c0bbbde4dca6fe877a1"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "1f0f50e01f3c7c5b9162b2ccdb8cae61"
server_key_iv: "e5e30427603f8b5d3d259e55ed2e460f"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.189954 5827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:24.201534 5827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:24.203370 5827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:24.203477 5827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:24.203642 5827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:24.203958 5827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:24.203996 5827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.204051 5827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:24.204078 5827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.209690 5827 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:35995
I20260923 22:17:24.209728 5954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:35995 every 8 connection(s)
I20260923 22:17:24.210014 5827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:24 dist-test-slave-dmw7 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201844, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:24.218794 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.210728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:38323 (local address 127.1.129.126:38227)
0923 22:17:24.210856 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.210858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.211002 (+ 144us) server_negotiation.cc:407] Connection header received
0923 22:17:24.211455 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.211456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.211466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.211524 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:24.212396 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.212890 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.213860 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.214060 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.214111 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.216103 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:24.216104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:24.216105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:24.216111 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:24.217555 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:24.217909 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:24.217910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:24.217910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:24.217971 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:24.218170 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:24.218170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:24.218171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:24.218268 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:24.218317 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.218610 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.218727 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260923 22:17:24.219017 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.210713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:38227 (local address 127.1.129.66:38323)
0923 22:17:24.210876 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.210917 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.211374 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.211581 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.211583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.211817 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:24.212305 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.212310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.212968 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.212969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.213767 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.213769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.214129 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.214130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.214224 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.214703 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:24.214711 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:24.216007 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:24.217630 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:24.217635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:24.217637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:24.217844 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:24.218026 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:24.218026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:24.218026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:24.218118 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:24.218352 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:24.218359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:24.218485 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.218590 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.218802 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260923 22:17:24.219581 5955 heartbeater.cc:347] Connected to a master server at 127.1.129.126:38227
I20260923 22:17:24.219681 5955 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:24.219755 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 5827
I20260923 22:17:24.219887 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:24.219882 5955 heartbeater.cc:511] Master 127.1.129.126:38227 requested a full tablet report, sending...
I20260923 22:17:24.220086 1541 external_mini_cluster.cc:1428] Setting key 35257aca35165671bb4898e6f1a6844b
I20260923 22:17:24.220309 5631 ts_manager.cc:194] Registered new tserver with Master: e9fc578742064c0bbbde4dca6fe877a1 (127.1.129.66:35995)
I20260923 22:17:24.220855 5631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:38323
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:24.246026 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:38227
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:24.324644 5961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:24.324851 5961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:24.324874 5961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:24.326108 5961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:24.326150 5961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:24.326186 5961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:24.327773 5961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:24.328016 5961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:24.328248 5961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:24.333321 5967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:24.333331 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.333371 5961 server_base.cc:1034] running on GCE node
W20260923 22:17:24.333328 5966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.333702 5961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:24.333945 5961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:24.335083 5961 hybrid_clock.cc:648] HybridClock initialized: now 1790201844335071 us; error 27 us; skew 500 ppm
Sep 23 22:17:24 dist-test-slave-dmw7 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201844, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:24.337483 5961 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:24.337813 5961 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:24.337863 5961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:24.338030 5961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:24.339123 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "b0009c7c3e2443d18de3defe78b2b0f8"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "26a210eb26761598e2345e5b1db5b532"
server_key_iv: "c9b575d3fbf3f342f0e77d4721ef1543"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.339491 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "b0009c7c3e2443d18de3defe78b2b0f8"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "26a210eb26761598e2345e5b1db5b532"
server_key_iv: "c9b575d3fbf3f342f0e77d4721ef1543"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.340821 5961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:24.341542 5975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.341830 5961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:24.341923 5961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "b0009c7c3e2443d18de3defe78b2b0f8"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "26a210eb26761598e2345e5b1db5b532"
server_key_iv: "c9b575d3fbf3f342f0e77d4721ef1543"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.341995 5961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:24.354449 5961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:24.356453 5961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:24.356606 5961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:24.356885 5961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:24.357204 5961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:24.357239 5961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.357261 5961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:24.357275 5961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.363221 5961 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:34035
I20260923 22:17:24.363314 6088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:34035 every 8 connection(s)
I20260923 22:17:24.363571 5961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:17:24 dist-test-slave-dmw7 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201844, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:24.370527 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 5961
I20260923 22:17:24.370613 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:24.370884 1541 external_mini_cluster.cc:1428] Setting key 0c883ac10c5c3fb2c81e7471379f9f18
I20260923 22:17:24.372790 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.364318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:39199 (local address 127.1.129.126:38227)
0923 22:17:24.364400 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.364402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.364618 (+ 216us) server_negotiation.cc:407] Connection header received
0923 22:17:24.365072 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.365073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.365081 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.365140 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:24.366044 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.366534 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.367501 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.367689 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.367742 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.369956 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:24.369957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:24.369958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:24.369964 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:24.371604 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:24.372021 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:24.372022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:24.372022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:24.372074 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:24.372251 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:24.372252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:24.372252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:24.372362 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:24.372448 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.372657 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.372727 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 22:17:24.372946 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.364309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:38227 (local address 127.1.129.67:39199)
0923 22:17:24.364481 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.364522 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.364993 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.365195 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.365198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.365457 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:24.365961 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.365966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.366617 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.366618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.367426 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.367428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.367771 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.367772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.367872 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.368443 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:24.368451 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:24.369857 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:24.371703 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:24.371709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:24.371712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:24.371961 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:24.372119 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:24.372119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:24.372120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:24.372203 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:24.372436 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:24.372439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:24.372543 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.372620 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.372733 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260923 22:17:24.373378 6089 heartbeater.cc:347] Connected to a master server at 127.1.129.126:38227
I20260923 22:17:24.373471 6089 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:24.373584 6089 heartbeater.cc:511] Master 127.1.129.126:38227 requested a full tablet report, sending...
I20260923 22:17:24.373932 5631 ts_manager.cc:194] Registered new tserver with Master: b0009c7c3e2443d18de3defe78b2b0f8 (127.1.129.67:34035)
I20260923 22:17:24.374303 5631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:39199
I20260923 22:17:24.382856 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:24.390056 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.384273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.1.129.126:38227)
0923 22:17:24.384337 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.384340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.384397 (+ 57us) server_negotiation.cc:407] Connection header received
0923 22:17:24.384439 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.384439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.384448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.384509 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:24.385045 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.385533 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.386240 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.386441 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.386503 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.387656 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:24.387657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:24.387658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:24.387664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:24.389149 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:24.389431 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:24.389432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:24.389432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:24.389483 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:24.389633 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:24.389634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:24.389634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:24.389766 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:24.389844 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.389938 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.389996 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:24.391599 5631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:24.393240 5631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:24.399958 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.395952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39477 (local address 127.0.0.1:59110)
0923 22:17:24.396151 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.396161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.396202 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.396411 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.396414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.396416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:24.396508 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.396510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.397871 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.397872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.398756 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.398758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.399656 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.399657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.399734 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.399749 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.399831 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.399880 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":40,"thread_start_us":63,"threads_started":1}
I20260923 22:17:24.399986 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.396061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.1.129.65:39477)
0923 22:17:24.396215 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.396221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.396231 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:24.396267 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.396271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.396286 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.396588 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:24.396685 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.397768 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.398841 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.399560 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.399636 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.399692 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.399856 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.399903 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260923 22:17:24.400161 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.395712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35995 (local address 127.0.0.1:34594)
0923 22:17:24.395915 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.395955 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.396022 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.396432 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.396433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.396434 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:24.396508 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.396511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.397819 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.397819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.399041 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.399043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.399951 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.399952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.400013 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.400018 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.400049 (+ 31us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.400091 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":64,"thread_start_us":86,"threads_started":1}
I20260923 22:17:24.400487 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.395816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.1.129.66:35995)
0923 22:17:24.396252 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.396256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.396264 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:24.396327 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.396329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.396343 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.396575 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:24.396655 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.397734 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.399126 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.399854 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.400268 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.400338 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.400365 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.400407 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":66,"threads_started":1}
I20260923 22:17:24.400723 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.395723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:34035 (local address 127.0.0.1:39186)
0923 22:17:24.396069 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.396077 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.396115 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.396532 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.396532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.396535 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:24.396582 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.396584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.397636 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.397637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.399306 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.399308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.400448 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.400449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.400511 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.400515 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.400572 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.400645 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":34,"thread_start_us":94,"threads_started":1}
I20260923 22:17:24.400970 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.395821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.1.129.67:34035)
0923 22:17:24.396345 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.396350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.396360 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:24.396403 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.396405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.396423 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.396668 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:24.396762 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.397505 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.399393 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.400342 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.400444 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.400498 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.400821 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.400872 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":82,"threads_started":1}
I20260923 22:17:24.400863 5755 tablet_service.cc:1467] Processing CreateTablet for tablet f3d5d2c8c2784265a73d23d4267f122f (DEFAULT_TABLE table=test-table [id=f895f4a21942455594ecf7b892c40f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:24.401235 5755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d5d2c8c2784265a73d23d4267f122f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:24.401571 6023 tablet_service.cc:1467] Processing CreateTablet for tablet f3d5d2c8c2784265a73d23d4267f122f (DEFAULT_TABLE table=test-table [id=f895f4a21942455594ecf7b892c40f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:24.401872 6023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d5d2c8c2784265a73d23d4267f122f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:24.403991 6107 tablet_bootstrap.cc:492] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e: Bootstrap starting.
I20260923 22:17:24.404639 6107 tablet_bootstrap.cc:654] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:24.404537 5889 tablet_service.cc:1467] Processing CreateTablet for tablet f3d5d2c8c2784265a73d23d4267f122f (DEFAULT_TABLE table=test-table [id=f895f4a21942455594ecf7b892c40f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:24.404762 6108 tablet_bootstrap.cc:492] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: Bootstrap starting.
I20260923 22:17:24.404868 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d5d2c8c2784265a73d23d4267f122f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:24.405040 6107 log.cc:824] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:24.405417 6108 tablet_bootstrap.cc:654] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:24.405826 6107 tablet_bootstrap.cc:492] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e: No bootstrap required, opened a new log
I20260923 22:17:24.405905 6107 ts_tablet_manager.cc:1397] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:24.405951 6108 log.cc:824] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:24.406713 6108 tablet_bootstrap.cc:492] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: No bootstrap required, opened a new log
I20260923 22:17:24.406844 6108 ts_tablet_manager.cc:1397] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:24.408429 6111 tablet_bootstrap.cc:492] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Bootstrap starting.
I20260923 22:17:24.408520 6108 raft_consensus.cc:348] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.408679 6108 raft_consensus.cc:374] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:24.408721 6108 raft_consensus.cc:729] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0009c7c3e2443d18de3defe78b2b0f8, State: Initialized, Role: FOLLOWER
I20260923 22:17:24.408696 6107 raft_consensus.cc:348] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.408869 6107 raft_consensus.cc:374] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:24.408941 6107 raft_consensus.cc:729] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5213eccc1d9412aade978ae9427263e, State: Initialized, Role: FOLLOWER
I20260923 22:17:24.408926 6108 consensus_queue.cc:260] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.409060 6111 tablet_bootstrap.cc:654] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:24.409067 6107 consensus_queue.cc:260] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.409339 6107 ts_tablet_manager.cc:1428] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:24.409358 6089 heartbeater.cc:503] Master 127.1.129.126:38227 was elected leader, sending a full tablet report...
I20260923 22:17:24.409348 6108 ts_tablet_manager.cc:1428] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:17:24.409646 6111 log.cc:824] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:24.410369 6111 tablet_bootstrap.cc:492] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: No bootstrap required, opened a new log
I20260923 22:17:24.410432 6111 ts_tablet_manager.cc:1397] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:24.411988 6111 raft_consensus.cc:348] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.412127 6111 raft_consensus.cc:374] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:24.412171 6111 raft_consensus.cc:729] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fc578742064c0bbbde4dca6fe877a1, State: Initialized, Role: FOLLOWER
I20260923 22:17:24.412292 6111 consensus_queue.cc:260] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.412530 6111 ts_tablet_manager.cc:1428] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:24.412590 5955 heartbeater.cc:503] Master 127.1.129.126:38227 was elected leader, sending a full tablet report...
W20260923 22:17:24.442308 5822 tablet.cc:2369] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:24.460546 5956 tablet.cc:2369] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:24.475623 6113 raft_consensus.cc:482] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:24.475728 6113 raft_consensus.cc:504] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.476104 6113 leader_election.cc:288] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9fc578742064c0bbbde4dca6fe877a1 (127.1.129.66:35995), b0009c7c3e2443d18de3defe78b2b0f8 (127.1.129.67:34035)
I20260923 22:17:24.479212 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.476418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:53277 (local address 127.1.129.67:34035)
0923 22:17:24.476499 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.476501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.476680 (+ 179us) server_negotiation.cc:407] Connection header received
0923 22:17:24.476706 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.476707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.476715 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.476767 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:24.476986 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.477456 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.478420 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.478857 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.478950 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.478958 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.479115 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.479148 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 22:17:24.479208 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.476316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:41257 (local address 127.1.129.66:35995)
0923 22:17:24.476380 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.476382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.476560 (+ 178us) server_negotiation.cc:407] Connection header received
0923 22:17:24.476600 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.476601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.476612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.476674 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:24.476860 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.477372 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.478419 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.478854 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.478951 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.478959 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.479100 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.479138 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:24.479223 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.476260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35995 (local address 127.1.129.65:41257)
0923 22:17:24.476465 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.476479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.476548 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.476691 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.476693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.476695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:24.476779 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.476782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.477473 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.477474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.478314 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.478317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.478950 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.478951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.479018 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.479029 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.479070 (+ 41us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.479127 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
I20260923 22:17:24.479233 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.476372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:34035 (local address 127.1.129.65:53277)
0923 22:17:24.476558 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.476568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.476629 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.476786 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.476786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.476788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:24.476884 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.476885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.477540 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.477541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.478324 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:24.478327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.478951 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.478952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:24.479030 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.479033 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.479103 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.479138 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":52,"thread_start_us":91,"threads_started":1}
I20260923 22:17:24.479508 5909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d5d2c8c2784265a73d23d4267f122f" candidate_uuid: "c5213eccc1d9412aade978ae9427263e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fc578742064c0bbbde4dca6fe877a1" is_pre_election: true
I20260923 22:17:24.479574 6043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d5d2c8c2784265a73d23d4267f122f" candidate_uuid: "c5213eccc1d9412aade978ae9427263e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" is_pre_election: true
I20260923 22:17:24.479661 5909 raft_consensus.cc:2454] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5213eccc1d9412aade978ae9427263e in term 0.
I20260923 22:17:24.479694 6043 raft_consensus.cc:2454] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5213eccc1d9412aade978ae9427263e in term 0.
I20260923 22:17:24.479924 5709 leader_election.cc:302] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5213eccc1d9412aade978ae9427263e, e9fc578742064c0bbbde4dca6fe877a1; no voters:
I20260923 22:17:24.480070 6113 raft_consensus.cc:2781] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:24.480147 6113 raft_consensus.cc:482] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:24.480176 6113 raft_consensus.cc:3037] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:24.480875 6113 raft_consensus.cc:504] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.481015 6113 leader_election.cc:288] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [CANDIDATE]: Term 1 election: Requested vote from peers e9fc578742064c0bbbde4dca6fe877a1 (127.1.129.66:35995), b0009c7c3e2443d18de3defe78b2b0f8 (127.1.129.67:34035)
I20260923 22:17:24.481146 5909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d5d2c8c2784265a73d23d4267f122f" candidate_uuid: "c5213eccc1d9412aade978ae9427263e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fc578742064c0bbbde4dca6fe877a1"
I20260923 22:17:24.481207 6043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d5d2c8c2784265a73d23d4267f122f" candidate_uuid: "c5213eccc1d9412aade978ae9427263e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0009c7c3e2443d18de3defe78b2b0f8"
I20260923 22:17:24.481235 5909 raft_consensus.cc:3037] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:24.481309 6043 raft_consensus.cc:3037] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:24.481983 5909 raft_consensus.cc:2454] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5213eccc1d9412aade978ae9427263e in term 1.
I20260923 22:17:24.481988 6043 raft_consensus.cc:2454] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5213eccc1d9412aade978ae9427263e in term 1.
I20260923 22:17:24.482149 5709 leader_election.cc:302] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5213eccc1d9412aade978ae9427263e, e9fc578742064c0bbbde4dca6fe877a1; no voters:
I20260923 22:17:24.482249 6113 raft_consensus.cc:2781] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:24.482379 6113 raft_consensus.cc:686] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [term 1 LEADER]: Becoming Leader. State: Replica: c5213eccc1d9412aade978ae9427263e, State: Running, Role: LEADER
I20260923 22:17:24.482486 6113 consensus_queue.cc:237] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } }
I20260923 22:17:24.483421 5631 catalog_manager.cc:5495] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e reported cstate change: term changed from 0 to 1, leader changed from <none> to c5213eccc1d9412aade978ae9427263e (127.1.129.65). New cstate: current_term: 1 leader_uuid: "c5213eccc1d9412aade978ae9427263e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5213eccc1d9412aade978ae9427263e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39477 } health_report { overall_health: HEALTHY } } }
I20260923 22:17:24.490482 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.488476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.1.129.65:39477)
0923 22:17:24.488517 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.488519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.488525 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:17:24.488595 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.488595 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.488609 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.488666 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:24.488775 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.489199 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.489897 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.490062 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.490109 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.490234 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:24.490317 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:24.490376 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.490407 (+ 31us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.490422 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:17:24.491981 6043 raft_consensus.cc:1261] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 1 FOLLOWER]: Refusing update from remote peer c5213eccc1d9412aade978ae9427263e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:24.491986 5909 raft_consensus.cc:1261] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 1 FOLLOWER]: Refusing update from remote peer c5213eccc1d9412aade978ae9427263e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:24.492277 6113 consensus_queue.cc:1035] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0009c7c3e2443d18de3defe78b2b0f8" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:24.492403 6118 consensus_queue.cc:1035] T f3d5d2c8c2784265a73d23d4267f122f P c5213eccc1d9412aade978ae9427263e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fc578742064c0bbbde4dca6fe877a1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:24.495466 6123 mvcc.cc:205] Tried to move back new op lower bound from 7332666755036979200 to 7332666755001401344. Current Snapshot: MvccSnapshot[applied={T|T < 7332666755036979200}]
I20260923 22:17:24.497247 5630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:24.497349 5630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:24.498039 5630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7da8c39985447c38fbde1ad7b05a881: Sending DeleteTablet for 3 replicas of tablet f3d5d2c8c2784265a73d23d4267f122f
I20260923 22:17:24.498379 6023 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d5d2c8c2784265a73d23d4267f122f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:24 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:39186
I20260923 22:17:24.498421 5889 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d5d2c8c2784265a73d23d4267f122f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:24 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:34594
I20260923 22:17:24.498591 5755 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d5d2c8c2784265a73d23d4267f122f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:24 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:59110
I20260923 22:17:24.498656 6131 tablet_replica.cc:330] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: stopping tablet replica
I20260923 22:17:24.498724 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 5693
I20260923 22:17:24.498752 6131 raft_consensus.cc:2229] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:24.498833 6131 raft_consensus.cc:2258] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:24.499984 6131 ts_tablet_manager.cc:1905] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:24.500522 6132 tablet_replica.cc:330] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: stopping tablet replica
I20260923 22:17:24.500643 6132 raft_consensus.cc:2229] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:24.500773 6132 raft_consensus.cc:2258] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:24.501144 6132 ts_tablet_manager.cc:1905] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:24.501858 6131 ts_tablet_manager.cc:1918] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:24.501943 6131 log.cc:1196] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/f3d5d2c8c2784265a73d23d4267f122f
I20260923 22:17:24.502200 6131 ts_tablet_manager.cc:1939] T f3d5d2c8c2784265a73d23d4267f122f P b0009c7c3e2443d18de3defe78b2b0f8: Deleting consensus metadata
I20260923 22:17:24.502478 6132 ts_tablet_manager.cc:1918] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:24.502571 6132 log.cc:1196] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/f3d5d2c8c2784265a73d23d4267f122f
I20260923 22:17:24.502656 5617 catalog_manager.cc:4841] TS b0009c7c3e2443d18de3defe78b2b0f8 (127.1.129.67:34035): tablet f3d5d2c8c2784265a73d23d4267f122f (table test-table [id=f895f4a21942455594ecf7b892c40f6a]) successfully deleted
I20260923 22:17:24.502929 6132 ts_tablet_manager.cc:1939] T f3d5d2c8c2784265a73d23d4267f122f P e9fc578742064c0bbbde4dca6fe877a1: Deleting consensus metadata
I20260923 22:17:24.503290 5616 catalog_manager.cc:4841] TS e9fc578742064c0bbbde4dca6fe877a1 (127.1.129.66:35995): tablet f3d5d2c8c2784265a73d23d4267f122f (table test-table [id=f895f4a21942455594ecf7b892c40f6a]) successfully deleted
W20260923 22:17:24.504303 5616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:39477 (error 108)
I20260923 22:17:24.504784 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.504676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39477 (local address 127.0.0.1:59132)
0923 22:17:24.504726 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.504760 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 22:17:24.504874 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 5827
W20260923 22:17:24.504937 5616 catalog_manager.cc:4584] TS c5213eccc1d9412aade978ae9427263e (127.1.129.65:39477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3d5d2c8c2784265a73d23d4267f122f: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39477: connect: Connection refused (error 111)
I20260923 22:17:24.509644 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 5961
I20260923 22:17:24.514001 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 5600
2026-09-23T22:17:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5999 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T22:17:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:24Z Disabled control of system clock
I20260923 22:17:24.541059 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:46655
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 22:17:24.620080 6138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:24.620280 6138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:24.620301 6138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:24.621366 6138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:24.621402 6138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:24.621414 6138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:24.621428 6138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:24.621439 6138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:24.622730 6138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:46655
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:46655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:24.622915 6138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:24.623135 6138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:24.627995 6146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:24.627966 6143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.627975 6138 server_base.cc:1034] running on GCE node
W20260923 22:17:24.627966 6144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.628453 6138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:24.628778 6138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:24.629940 6138 hybrid_clock.cc:648] HybridClock initialized: now 1790201844629929 us; error 42 us; skew 500 ppm
I20260923 22:17:24.631167 6138 webserver.cc:460] Webserver started at http://127.1.129.126:32835/ using document root <none> and password file <none>
I20260923 22:17:24.631417 6138 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:24.631484 6138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:24.631691 6138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:24.632745 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "7095c9cc888716dbb60797d68f467845"
server_key_iv: "645b8f0e3624602dcb7a521f2deaa481"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.633100 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "7095c9cc888716dbb60797d68f467845"
server_key_iv: "645b8f0e3624602dcb7a521f2deaa481"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.634411 6138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:24.635249 6152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.635478 6138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:24.635573 6138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "7095c9cc888716dbb60797d68f467845"
server_key_iv: "645b8f0e3624602dcb7a521f2deaa481"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.635654 6138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:24.638936 6138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:24.639195 6138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:24.639346 6138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:24.643034 6138 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:46655
I20260923 22:17:24.643065 6204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:46655 every 8 connection(s)
I20260923 22:17:24.643384 6138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:24.643988 6205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:24.644744 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 6138
I20260923 22:17:24.644827 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:24.645006 1541 external_mini_cluster.cc:1428] Setting key 5abfe3e6a2ad3cf19c2dbdfca56c526f
I20260923 22:17:24.647656 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.645519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.1.129.126:46655)
0923 22:17:24.645731 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.645740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.645775 (+ 35us) server_negotiation.cc:407] Connection header received
0923 22:17:24.646231 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.646247 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.646301 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.646452 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:24.647132 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:24.647135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:24.647136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:24.647139 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:24.647179 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:24.647192 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:24.647287 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.647323 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.647477 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260923 22:17:24.648039 6205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d: Bootstrap starting.
I20260923 22:17:24.648679 6205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:24.648908 6205 log.cc:824] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:24.649555 6205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d: No bootstrap required, opened a new log
I20260923 22:17:24.650998 6205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46655 } }
I20260923 22:17:24.651154 6205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:24.651180 6205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d94ac62960f44f6a1b4e0a79a86ef6d, State: Initialized, Role: FOLLOWER
I20260923 22:17:24.651289 6205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46655 } }
I20260923 22:17:24.651372 6205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:24.651394 6205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:24.651422 6205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:24.652237 6205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46655 } }
I20260923 22:17:24.652326 6205 leader_election.cc:302] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d94ac62960f44f6a1b4e0a79a86ef6d; no voters:
I20260923 22:17:24.652442 6205 leader_election.cc:288] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:24.652493 6210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:24.652628 6210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [term 1 LEADER]: Becoming Leader. State: Replica: 2d94ac62960f44f6a1b4e0a79a86ef6d, State: Running, Role: LEADER
I20260923 22:17:24.652683 6205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:24.652762 6210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46655 } }
I20260923 22:17:24.653175 6211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46655 } } }
I20260923 22:17:24.653317 6211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:24.653187 6212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d94ac62960f44f6a1b4e0a79a86ef6d. Latest consensus state: current_term: 1 leader_uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d94ac62960f44f6a1b4e0a79a86ef6d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46655 } } }
I20260923 22:17:24.653416 6212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:24.654392 6226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:24.654449 6226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:24.654623 6219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:24.654850 6219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:24.656283 6219 catalog_manager.cc:1347] Generated new cluster ID: 8e7a9d34057c4b70b4427ea07ce29be3
I20260923 22:17:24.656342 6219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:24.678534 6219 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:24.678789 6219 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:24.695912 6219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d: Generated new TSK 0
I20260923 22:17:24.696165 6219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:24.700467 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46655
--builtin_ntp_servers=127.1.129.84:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 22:17:24.785851 6229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:24.786051 6229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:24.786072 6229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:24.787266 6229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:24.787305 6229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:24.787333 6229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:24.788669 6229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:24.788913 6229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:24.789212 6229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:24.794090 6235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:24.794126 6237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.794265 6229 server_base.cc:1034] running on GCE node
W20260923 22:17:24.794127 6234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.794556 6229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:24.794842 6229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:24.795985 6229 hybrid_clock.cc:648] HybridClock initialized: now 1790201844795978 us; error 29 us; skew 500 ppm
I20260923 22:17:24.797253 6229 webserver.cc:460] Webserver started at http://127.1.129.65:39731/ using document root <none> and password file <none>
I20260923 22:17:24.797501 6229 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:24.797547 6229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:24.797713 6229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:24.798815 6229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "05caa6f9009b4c4cbae8976a6454ed6e"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "fba1f8f92922c51edfeb057dfa07e660"
server_key_iv: "d1697655cf76b7199e52985ad1dd16f4"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.799162 6229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "05caa6f9009b4c4cbae8976a6454ed6e"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "fba1f8f92922c51edfeb057dfa07e660"
server_key_iv: "d1697655cf76b7199e52985ad1dd16f4"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.800411 6229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:24.801069 6243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.801229 6229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.801344 6229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "05caa6f9009b4c4cbae8976a6454ed6e"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "fba1f8f92922c51edfeb057dfa07e660"
server_key_iv: "d1697655cf76b7199e52985ad1dd16f4"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.801422 6229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:24.804733 6229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:24.804980 6229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:24.805140 6229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:24.805348 6229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:24.805634 6229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:24.805686 6229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.805725 6229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:24.805763 6229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.811349 6229 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:45565
I20260923 22:17:24.811395 6356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:45565 every 8 connection(s)
I20260923 22:17:24.811674 6229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:24.814450 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.812331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:41019 (local address 127.1.129.126:46655)
0923 22:17:24.812372 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.812374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.812592 (+ 218us) server_negotiation.cc:407] Connection header received
0923 22:17:24.813085 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.813086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.813093 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.813157 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:24.814220 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:24.814221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:24.814222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:24.814227 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:24.814248 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:24.814252 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:24.814302 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.814371 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.814385 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:17:24.814678 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.812299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46655 (local address 127.1.129.65:41019)
0923 22:17:24.812474 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.812505 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.812980 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.813221 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.813225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.813235 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:24.814078 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:24.814078 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:24.814103 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:24.814129 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:24.814299 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:24.814303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:24.814306 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.814380 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.814436 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260923 22:17:24.815135 6357 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46655
I20260923 22:17:24.815212 6357 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:24.815357 6357 heartbeater.cc:511] Master 127.1.129.126:46655 requested a full tablet report, sending...
I20260923 22:17:24.815377 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 6229
I20260923 22:17:24.815497 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:24.815687 1541 external_mini_cluster.cc:1428] Setting key d18bd2d30308ef34f5c12f57d02dcc4a
I20260923 22:17:24.815922 6169 ts_manager.cc:194] Registered new tserver with Master: 05caa6f9009b4c4cbae8976a6454ed6e (127.1.129.65:45565)
I20260923 22:17:24.816687 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46655
--builtin_ntp_servers=127.1.129.84:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 22:17:24.896255 6360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:24.896459 6360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:24.896481 6360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:24.897562 6360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:24.897593 6360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:24.897617 6360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:24.898895 6360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:24.899073 6360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:24.899320 6360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:24.904062 6360 server_base.cc:1034] running on GCE node
W20260923 22:17:24.904121 6366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:24.904034 6365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:24.904043 6368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.904393 6360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:24.904680 6360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:24.905855 6360 hybrid_clock.cc:648] HybridClock initialized: now 1790201844905848 us; error 19 us; skew 500 ppm
I20260923 22:17:24.907059 6360 webserver.cc:460] Webserver started at http://127.1.129.66:43011/ using document root <none> and password file <none>
I20260923 22:17:24.907330 6360 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:24.907409 6360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:24.907544 6360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:24.908565 6360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "ba8360f005fc474abf4f9242539d1927"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "1a7c5bbb6c21bae3f8b44f5cfa78cefd"
server_key_iv: "71f2d16a2da7df395a90ad9aae181cbb"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.908933 6360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "ba8360f005fc474abf4f9242539d1927"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "1a7c5bbb6c21bae3f8b44f5cfa78cefd"
server_key_iv: "71f2d16a2da7df395a90ad9aae181cbb"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.910171 6360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:24.910890 6374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.911055 6360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:24.911152 6360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "ba8360f005fc474abf4f9242539d1927"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-dmw7"
server_key: "1a7c5bbb6c21bae3f8b44f5cfa78cefd"
server_key_iv: "71f2d16a2da7df395a90ad9aae181cbb"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.911229 6360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:24.914407 6360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:24.914708 6360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:24.914870 6360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:24.915074 6360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:24.915378 6360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:24.915432 6360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.915472 6360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:24.915498 6360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.921077 6360 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:32801
I20260923 22:17:24.921121 6487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:32801 every 8 connection(s)
I20260923 22:17:24.921422 6360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:24.924393 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.922083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:33175 (local address 127.1.129.126:46655)
0923 22:17:24.922125 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.922127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.922383 (+ 256us) server_negotiation.cc:407] Connection header received
0923 22:17:24.922948 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.922949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.922955 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.923011 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:24.924120 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:24.924121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:24.924122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:24.924127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:24.924141 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:24.924145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:24.924200 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.924301 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.924315 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:24.924577 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.922084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46655 (local address 127.1.129.66:33175)
0923 22:17:24.922262 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:24.922292 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:24.922836 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:24.923087 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:24.923090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:24.923098 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:24.923959 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:24.923960 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:24.923987 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:24.924029 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:24.924211 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:24.924216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:24.924218 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:24.924311 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 22:17:24.924371 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260923 22:17:24.925055 6488 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46655
I20260923 22:17:24.925123 6488 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:24.925276 6488 heartbeater.cc:511] Master 127.1.129.126:46655 requested a full tablet report, sending...
I20260923 22:17:24.925621 6168 ts_manager.cc:194] Registered new tserver with Master: ba8360f005fc474abf4f9242539d1927 (127.1.129.66:32801)
I20260923 22:17:24.931037 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 6360
I20260923 22:17:24.931123 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:24.931252 1541 external_mini_cluster.cc:1428] Setting key 30567191460b90c9d29e6576d052e4d7
I20260923 22:17:24.932144 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46655
--builtin_ntp_servers=127.1.129.84:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 22:17:25.012293 6491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:25.012550 6491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:25.012589 6491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:25.013842 6491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:25.013906 6491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:25.013947 6491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:25.015486 6491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:25.015772 6491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:25.016081 6491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:25.021056 6496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:25.021044 6497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.021136 6491 server_base.cc:1034] running on GCE node
W20260923 22:17:25.021047 6499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.021463 6491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:25.021710 6491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:25.022881 6491 hybrid_clock.cc:648] HybridClock initialized: now 1790201845022867 us; error 28 us; skew 500 ppm
I20260923 22:17:25.024230 6491 webserver.cc:460] Webserver started at http://127.1.129.67:41603/ using document root <none> and password file <none>
I20260923 22:17:25.024487 6491 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:25.024562 6491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:25.024739 6491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:25.025792 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "869b7cac70644c4c91a8c6e412835a71"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "d7655020b8e31557514619aded82eb7b"
server_key_iv: "292c063e397518e224f0c73283649494"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.026162 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "869b7cac70644c4c91a8c6e412835a71"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "d7655020b8e31557514619aded82eb7b"
server_key_iv: "292c063e397518e224f0c73283649494"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.027570 6491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:25.028335 6505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:25.028517 6491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:25.028628 6491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "869b7cac70644c4c91a8c6e412835a71"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "d7655020b8e31557514619aded82eb7b"
server_key_iv: "292c063e397518e224f0c73283649494"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.028703 6491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:25.032548 6491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:25.032795 6491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:25.032928 6491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:25.033152 6491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:25.033460 6491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:25.033514 6491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:25.033553 6491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:25.033582 6491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:25.039330 6491 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:38355
I20260923 22:17:25.039402 6618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:38355 every 8 connection(s)
I20260923 22:17:25.039728 6491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:25.042608 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.040508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:44933 (local address 127.1.129.126:46655)
0923 22:17:25.040565 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.040567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.040855 (+ 288us) server_negotiation.cc:407] Connection header received
0923 22:17:25.041297 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.041298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.041303 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.041356 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.042422 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.042423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.042423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.042428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.042440 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.042444 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.042484 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.042546 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.042558 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:17:25.042858 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.040503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46655 (local address 127.1.129.67:44933)
0923 22:17:25.040688 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.040755 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.041209 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.041408 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.041410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.041418 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:25.042294 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:25.042294 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:25.042320 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:25.042347 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:25.042489 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:25.042492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:25.042494 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:25.042542 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:17:25.042585 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260923 22:17:25.043322 6619 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46655
I20260923 22:17:25.043427 6619 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:25.043567 6619 heartbeater.cc:511] Master 127.1.129.126:46655 requested a full tablet report, sending...
I20260923 22:17:25.043931 6168 ts_manager.cc:194] Registered new tserver with Master: 869b7cac70644c4c91a8c6e412835a71 (127.1.129.67:38355)
I20260923 22:17:25.046471 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 6491
I20260923 22:17:25.046554 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:25.046753 1541 external_mini_cluster.cc:1428] Setting key fd4f7a0a92c93f7d7b6c3387c7a8c151
I20260923 22:17:25.047505 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:25.049698 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.048755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.1.129.126:46655)
0923 22:17:25.048810 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.048812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.048946 (+ 134us) server_negotiation.cc:407] Connection header received
0923 22:17:25.048992 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.048993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.048996 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.049043 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.049548 (+ 505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.049549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.049550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.049553 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.049563 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.049567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.049608 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.049642 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.049652 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:17:25.051340 6168 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:25.052912 6168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:25.056950 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.055111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:38355 (local address 127.0.0.1:50100)
0923 22:17:25.055298 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.055332 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.055387 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.056254 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.056254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.056256 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:25.056561 (+ 305us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:25.056562 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:25.056566 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:25.056576 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:25.056781 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:25.056787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:25.056789 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:25.056869 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:17:25.056883 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260923 22:17:25.056978 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.055173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.1.129.67:38355)
0923 22:17:25.055615 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.055621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.055629 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:25.055650 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.055652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.055659 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.055997 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.056647 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.056653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.056654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.056659 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.056692 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.056707 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.056772 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.056887 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.056901 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":72,"threads_started":1}
I20260923 22:17:25.057093 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.055358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.1.129.65:45565)
0923 22:17:25.055543 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.055548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.055559 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:25.055587 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.055589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.055599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.055670 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.056717 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.056725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.056725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.056730 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.056763 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.056790 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.056862 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.056983 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.056999 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260923 22:17:25.057111 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.055123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:32801 (local address 127.0.0.1:35940)
0923 22:17:25.055670 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.055681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.055730 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.055937 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.055943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.055949 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:25.056725 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:25.056726 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:25.056733 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:25.056745 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:25.056999 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:25.056999 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:25.057000 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:25.057040 (+ 40us) client_negotiation.cc:240] Negotiation successful
0923 22:17:25.057050 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":88,"threads_started":1}
I20260923 22:17:25.057288 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.055501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.1.129.66:32801)
0923 22:17:25.055649 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.055716 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.055726 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:25.055834 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.055837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.055846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.055937 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.056857 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.056864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.056865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.056869 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.056899 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.056924 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.056975 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.057183 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.057200 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260923 22:17:25.057669 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.055112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:45565 (local address 127.0.0.1:45694)
0923 22:17:25.055469 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.055479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.055512 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.055995 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.055996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.055999 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:25.056447 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:25.056448 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:25.056482 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:25.056511 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:25.056888 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:25.056889 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:25.056890 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:25.056934 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 22:17:25.056947 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260923 22:17:25.057993 6422 tablet_service.cc:1467] Processing CreateTablet for tablet 2a98349765564fd3811fe8d55d8225ba (DEFAULT_TABLE table=test-table [id=98afb9503fb443229fe78c69eac51684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:25.057997 6553 tablet_service.cc:1467] Processing CreateTablet for tablet 2a98349765564fd3811fe8d55d8225ba (DEFAULT_TABLE table=test-table [id=98afb9503fb443229fe78c69eac51684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:25.058336 6553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a98349765564fd3811fe8d55d8225ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:25.058352 6422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a98349765564fd3811fe8d55d8225ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:25.058416 6291 tablet_service.cc:1467] Processing CreateTablet for tablet 2a98349765564fd3811fe8d55d8225ba (DEFAULT_TABLE table=test-table [id=98afb9503fb443229fe78c69eac51684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:25.058730 6291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a98349765564fd3811fe8d55d8225ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:25.060990 6633 tablet_bootstrap.cc:492] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: Bootstrap starting.
I20260923 22:17:25.061699 6634 tablet_bootstrap.cc:492] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: Bootstrap starting.
I20260923 22:17:25.061630 6633 tablet_bootstrap.cc:654] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:25.061856 6635 tablet_bootstrap.cc:492] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e: Bootstrap starting.
I20260923 22:17:25.062147 6633 log.cc:824] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:25.062351 6634 tablet_bootstrap.cc:654] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:25.062536 6635 tablet_bootstrap.cc:654] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:25.062848 6635 log.cc:824] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:25.062852 6634 log.cc:824] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:25.063237 6633 tablet_bootstrap.cc:492] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: No bootstrap required, opened a new log
I20260923 22:17:25.063328 6633 ts_tablet_manager.cc:1397] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:25.063608 6635 tablet_bootstrap.cc:492] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e: No bootstrap required, opened a new log
I20260923 22:17:25.063678 6635 ts_tablet_manager.cc:1397] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:17:25.066008 6633 raft_consensus.cc:348] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.066203 6633 raft_consensus.cc:374] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:25.066279 6633 raft_consensus.cc:729] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8360f005fc474abf4f9242539d1927, State: Initialized, Role: FOLLOWER
I20260923 22:17:25.066207 6635 raft_consensus.cc:348] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.066339 6635 raft_consensus.cc:374] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:25.066378 6635 raft_consensus.cc:729] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05caa6f9009b4c4cbae8976a6454ed6e, State: Initialized, Role: FOLLOWER
I20260923 22:17:25.066407 6633 consensus_queue.cc:260] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.066468 6635 consensus_queue.cc:260] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.066735 6635 ts_tablet_manager.cc:1428] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:17:25.066740 6633 ts_tablet_manager.cc:1428] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260923 22:17:25.066779 6357 heartbeater.cc:503] Master 127.1.129.126:46655 was elected leader, sending a full tablet report...
I20260923 22:17:25.066864 6488 heartbeater.cc:503] Master 127.1.129.126:46655 was elected leader, sending a full tablet report...
I20260923 22:17:25.067134 6634 tablet_bootstrap.cc:492] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: No bootstrap required, opened a new log
I20260923 22:17:25.067246 6634 ts_tablet_manager.cc:1397] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260923 22:17:25.069234 6634 raft_consensus.cc:348] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.069353 6634 raft_consensus.cc:374] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:25.069373 6634 raft_consensus.cc:729] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b7cac70644c4c91a8c6e412835a71, State: Initialized, Role: FOLLOWER
I20260923 22:17:25.069448 6634 consensus_queue.cc:260] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.069624 6634 ts_tablet_manager.cc:1428] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:17:25.069672 6619 heartbeater.cc:503] Master 127.1.129.126:46655 was elected leader, sending a full tablet report...
I20260923 22:17:25.115065 6639 raft_consensus.cc:482] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:25.115182 6639 raft_consensus.cc:504] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.115477 6639 leader_election.cc:288] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869b7cac70644c4c91a8c6e412835a71 (127.1.129.67:38355), ba8360f005fc474abf4f9242539d1927 (127.1.129.66:32801)
I20260923 22:17:25.116886 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.115663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:38355 (local address 127.1.129.65:35439)
0923 22:17:25.115721 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.115735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.115793 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.115970 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.115971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.115973 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:25.116487 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:25.116488 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:25.116496 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:25.116511 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:25.116704 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:25.116705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:25.116705 (+ 0us) client_negotiation.cc:769] Sending connection context
0923 22:17:25.116788 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 22:17:25.116802 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 22:17:25.116925 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.115803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:35439 (local address 127.1.129.67:38355)
0923 22:17:25.115846 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.115848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.115855 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:25.115878 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.115879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.115884 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.115949 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.116602 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.116604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.116604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.116611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.116632 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.116638 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.116702 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.116801 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.116821 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:25.117020 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.115663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:32801 (local address 127.1.129.65:42677)
0923 22:17:25.115872 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.115880 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.115948 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.116074 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.116075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.116089 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:25.116664 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:25.116665 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:25.116672 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:25.116685 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:25.116872 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:25.116873 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:25.116874 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:25.116925 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:17:25.116938 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":20,"thread_start_us":104,"threads_started":1}
I20260923 22:17:25.117018 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.115748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:42677 (local address 127.1.129.66:32801)
0923 22:17:25.115812 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.115814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.115944 (+ 130us) server_negotiation.cc:407] Connection header received
0923 22:17:25.116005 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.116006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.116011 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.116065 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.116773 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.116774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.116775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.116779 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.116798 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.116802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.116857 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.116924 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.116938 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:25.117297 6573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a98349765564fd3811fe8d55d8225ba" candidate_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b7cac70644c4c91a8c6e412835a71" is_pre_election: true
I20260923 22:17:25.117341 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a98349765564fd3811fe8d55d8225ba" candidate_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8360f005fc474abf4f9242539d1927" is_pre_election: true
I20260923 22:17:25.117420 6573 raft_consensus.cc:2454] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05caa6f9009b4c4cbae8976a6454ed6e in term 0.
I20260923 22:17:25.117491 6442 raft_consensus.cc:2454] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05caa6f9009b4c4cbae8976a6454ed6e in term 0.
I20260923 22:17:25.117657 6245 leader_election.cc:302] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05caa6f9009b4c4cbae8976a6454ed6e, 869b7cac70644c4c91a8c6e412835a71; no voters:
I20260923 22:17:25.117777 6639 raft_consensus.cc:2781] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:25.117847 6639 raft_consensus.cc:482] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:25.117865 6639 raft_consensus.cc:3037] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:25.118512 6639 raft_consensus.cc:504] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.118673 6639 leader_election.cc:288] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [CANDIDATE]: Term 1 election: Requested vote from peers 869b7cac70644c4c91a8c6e412835a71 (127.1.129.67:38355), ba8360f005fc474abf4f9242539d1927 (127.1.129.66:32801)
I20260923 22:17:25.118774 6573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a98349765564fd3811fe8d55d8225ba" candidate_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b7cac70644c4c91a8c6e412835a71"
I20260923 22:17:25.118806 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a98349765564fd3811fe8d55d8225ba" candidate_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8360f005fc474abf4f9242539d1927"
I20260923 22:17:25.118872 6442 raft_consensus.cc:3037] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:25.118875 6573 raft_consensus.cc:3037] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:25.119549 6442 raft_consensus.cc:2454] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05caa6f9009b4c4cbae8976a6454ed6e in term 1.
I20260923 22:17:25.119567 6573 raft_consensus.cc:2454] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05caa6f9009b4c4cbae8976a6454ed6e in term 1.
I20260923 22:17:25.119715 6245 leader_election.cc:302] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05caa6f9009b4c4cbae8976a6454ed6e, 869b7cac70644c4c91a8c6e412835a71; no voters:
I20260923 22:17:25.119791 6639 raft_consensus.cc:2781] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:25.119967 6639 raft_consensus.cc:686] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [term 1 LEADER]: Becoming Leader. State: Replica: 05caa6f9009b4c4cbae8976a6454ed6e, State: Running, Role: LEADER
I20260923 22:17:25.120093 6639 consensus_queue.cc:237] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } }
I20260923 22:17:25.120951 6169 catalog_manager.cc:5495] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05caa6f9009b4c4cbae8976a6454ed6e (127.1.129.65). New cstate: current_term: 1 leader_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05caa6f9009b4c4cbae8976a6454ed6e" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:25.129741 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.128814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.1.129.65:45565)
0923 22:17:25.128878 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.128881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.128890 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:25.128962 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.128963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.128975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.129023 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.129528 (+ 505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.129529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.129529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.129534 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.129548 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.129552 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.129608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.129654 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.129669 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:25.131583 6573 raft_consensus.cc:1261] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 1 FOLLOWER]: Refusing update from remote peer 05caa6f9009b4c4cbae8976a6454ed6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:25.131583 6442 raft_consensus.cc:1261] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 1 FOLLOWER]: Refusing update from remote peer 05caa6f9009b4c4cbae8976a6454ed6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:25.131951 6639 consensus_queue.cc:1035] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8360f005fc474abf4f9242539d1927" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:25.132097 6643 consensus_queue.cc:1035] T 2a98349765564fd3811fe8d55d8225ba P 05caa6f9009b4c4cbae8976a6454ed6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "869b7cac70644c4c91a8c6e412835a71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:25.134411 6646 mvcc.cc:205] Tried to move back new op lower bound from 7332666757655961600 to 7332666757612634112. Current Snapshot: MvccSnapshot[applied={T|T < 7332666757655961600}]
I20260923 22:17:25.134714 6648 mvcc.cc:205] Tried to move back new op lower bound from 7332666757655961600 to 7332666757612634112. Current Snapshot: MvccSnapshot[applied={T|T < 7332666757655961600}]
I20260923 22:17:25.134953 6647 mvcc.cc:205] Tried to move back new op lower bound from 7332666757655961600 to 7332666757612634112. Current Snapshot: MvccSnapshot[applied={T|T < 7332666757655961600}]
I20260923 22:17:25.136509 6169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:25.136603 6169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:25.137287 6169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d94ac62960f44f6a1b4e0a79a86ef6d: Sending DeleteTablet for 3 replicas of tablet 2a98349765564fd3811fe8d55d8225ba
I20260923 22:17:25.137614 6291 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a98349765564fd3811fe8d55d8225ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:25 UTC) from {username='slave'} at 127.0.0.1:45694
I20260923 22:17:25.137656 6422 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a98349765564fd3811fe8d55d8225ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:25 UTC) from {username='slave'} at 127.0.0.1:35940
I20260923 22:17:25.137931 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 6229
I20260923 22:17:25.137933 6553 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a98349765564fd3811fe8d55d8225ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:25 UTC) from {username='slave'} at 127.0.0.1:50100
I20260923 22:17:25.139194 6657 tablet_replica.cc:330] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: stopping tablet replica
I20260923 22:17:25.139307 6657 raft_consensus.cc:2229] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:25.139415 6657 raft_consensus.cc:2258] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:25.139792 6657 ts_tablet_manager.cc:1905] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:25.141223 6657 ts_tablet_manager.cc:1918] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:25.141323 6657 log.cc:1196] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/2a98349765564fd3811fe8d55d8225ba
I20260923 22:17:25.141606 6657 ts_tablet_manager.cc:1939] T 2a98349765564fd3811fe8d55d8225ba P ba8360f005fc474abf4f9242539d1927: Deleting consensus metadata
I20260923 22:17:25.141979 6155 catalog_manager.cc:4841] TS ba8360f005fc474abf4f9242539d1927 (127.1.129.66:32801): tablet 2a98349765564fd3811fe8d55d8225ba (table test-table [id=98afb9503fb443229fe78c69eac51684]) successfully deleted
W20260923 22:17:25.143967 6153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:45565 (error 108)
I20260923 22:17:25.144474 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.144306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:45565 (local address 127.0.0.1:45706)
0923 22:17:25.144368 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.144438 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 22:17:25.144508 6658 tablet_replica.cc:330] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: stopping tablet replica
I20260923 22:17:25.144636 6658 raft_consensus.cc:2229] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260923 22:17:25.144613 6153 catalog_manager.cc:4584] TS 05caa6f9009b4c4cbae8976a6454ed6e (127.1.129.65:45565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a98349765564fd3811fe8d55d8225ba: Network error: Client connection negotiation failed: client connection to 127.1.129.65:45565: connect: Connection refused (error 111)
I20260923 22:17:25.144791 6658 raft_consensus.cc:2258] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:25.144968 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 6360
I20260923 22:17:25.145630 6658 ts_tablet_manager.cc:1905] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:25.147033 6658 ts_tablet_manager.cc:1918] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:25.147133 6658 log.cc:1196] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/2a98349765564fd3811fe8d55d8225ba
I20260923 22:17:25.147414 6658 ts_tablet_manager.cc:1939] T 2a98349765564fd3811fe8d55d8225ba P 869b7cac70644c4c91a8c6e412835a71: Deleting consensus metadata
I20260923 22:17:25.147843 6154 catalog_manager.cc:4841] TS 869b7cac70644c4c91a8c6e412835a71 (127.1.129.67:38355): tablet 2a98349765564fd3811fe8d55d8225ba (table test-table [id=98afb9503fb443229fe78c69eac51684]) successfully deleted
I20260923 22:17:25.150115 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 6491
I20260923 22:17:25.155717 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 6138
2026-09-23T22:17:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (640 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 22:17:25.174901 1541 mini_oidc.cc:152] Starting JWKS server
I20260923 22:17:25.175957 1541 webserver.cc:460] Webserver started at https://127.0.0.1:40859/ using document root <none> and password file <none>
I20260923 22:17:25.176033 1541 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 22:17:25.176156 1541 webserver.cc:460] Webserver started at http://127.0.0.1:34393/ using document root <none> and password file <none>
2026-09-23T22:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:25Z Disabled control of system clock
I20260923 22:17:25.183779 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44893
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:44893
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40859/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/testchainca.cert with env {}
W20260923 22:17:25.262245 6665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:25.262506 6665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:25.262543 6665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:25.263769 6665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:25.263850 6665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:25.263873 6665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:25.263893 6665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:25.263924 6665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 22:17:25.263947 6665 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 22:17:25.263968 6665 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40859/jwks.json
I20260923 22:17:25.265517 6665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:44893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40859/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:25.265909 6665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:25.266141 6665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:25.270906 6671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:25.270923 6670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:25.270923 6673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.271211 6665 server_base.cc:1034] running on GCE node
I20260923 22:17:25.271477 6665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:25.271765 6665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:25.272933 6665 hybrid_clock.cc:648] HybridClock initialized: now 1790201845272912 us; error 34 us; skew 500 ppm
I20260923 22:17:25.274468 6665 webserver.cc:460] Webserver started at http://127.1.129.126:36423/ using document root <none> and password file <none>
I20260923 22:17:25.274789 6665 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:25.274866 6665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:25.275000 6665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:25.276134 6665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "e3488ec85ecf42b2b20874ee12b6f558"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "f08ea5ff6b2768fd26a49a3554fe39f6"
server_key_iv: "2c4a9c8e7a08981c951214897bf62310"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.276513 6665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "e3488ec85ecf42b2b20874ee12b6f558"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "f08ea5ff6b2768fd26a49a3554fe39f6"
server_key_iv: "2c4a9c8e7a08981c951214897bf62310"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.277825 6665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:25.278510 6679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:25.278728 6665 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:25.278827 6665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "e3488ec85ecf42b2b20874ee12b6f558"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "f08ea5ff6b2768fd26a49a3554fe39f6"
server_key_iv: "2c4a9c8e7a08981c951214897bf62310"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.278914 6665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:25.299562 6665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:25.299891 6665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:25.300051 6665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:25.303913 6665 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:44893
I20260923 22:17:25.303961 6731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:44893 every 8 connection(s)
I20260923 22:17:25.304318 6665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:25.305285 6732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:25.307580 6732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558: Bootstrap starting.
I20260923 22:17:25.308184 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 6665
I20260923 22:17:25.308281 6732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:25.308288 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:25.308502 1541 external_mini_cluster.cc:1428] Setting key daa48fd5410d42d70c8eb01f7ed413dc
I20260923 22:17:25.308606 6732 log.cc:824] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:25.309408 6732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558: No bootstrap required, opened a new log
I20260923 22:17:25.311632 6732 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3488ec85ecf42b2b20874ee12b6f558" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44893 } }
I20260923 22:17:25.311797 6732 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:25.311841 6732 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3488ec85ecf42b2b20874ee12b6f558, State: Initialized, Role: FOLLOWER
I20260923 22:17:25.311961 6732 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3488ec85ecf42b2b20874ee12b6f558" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44893 } }
I20260923 22:17:25.312029 6732 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:25.312057 6732 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:25.312111 6732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:25.312719 6732 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3488ec85ecf42b2b20874ee12b6f558" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44893 } }
I20260923 22:17:25.312844 6732 leader_election.cc:302] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3488ec85ecf42b2b20874ee12b6f558; no voters:
I20260923 22:17:25.313001 6732 leader_election.cc:288] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:25.313040 6737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:25.313220 6737 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [term 1 LEADER]: Becoming Leader. State: Replica: e3488ec85ecf42b2b20874ee12b6f558, State: Running, Role: LEADER
I20260923 22:17:25.313309 6732 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:25.313374 6737 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3488ec85ecf42b2b20874ee12b6f558" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44893 } }
I20260923 22:17:25.313977 6739 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3488ec85ecf42b2b20874ee12b6f558. Latest consensus state: current_term: 1 leader_uuid: "e3488ec85ecf42b2b20874ee12b6f558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3488ec85ecf42b2b20874ee12b6f558" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44893 } } }
I20260923 22:17:25.314128 6739 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:25.314173 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.309033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.1.129.126:44893)
0923 22:17:25.309209 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.309217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.309243 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:17:25.309683 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.309690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.309754 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.309868 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.310123 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.311016 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.312148 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.312575 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.312663 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.313403 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.313407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.313407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.313609 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.313652 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.313666 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.313735 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.313839 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.313921 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260923 22:17:25.314456 6738 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3488ec85ecf42b2b20874ee12b6f558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3488ec85ecf42b2b20874ee12b6f558" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44893 } } }
I20260923 22:17:25.314580 6738 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:25.315224 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 22:17:25.315466 6753 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:25.315521 6753 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:25.315572 6746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:25.315712 6746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:25.317168 6746 catalog_manager.cc:1347] Generated new cluster ID: 521f13870fa8407fa8b27f8b7ec84fca
I20260923 22:17:25.317226 6746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:25.322963 6746 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:25.323447 6746 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:25.329519 6746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3488ec85ecf42b2b20874ee12b6f558: Generated new TSK 0
I20260923 22:17:25.329728 6746 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 22:17:25.353411 6762 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 22:17:25.353528 6762 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260923 22:17:25.354130 6761 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:25.337969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44893 (local address 127.0.0.1:46802)
0923 22:17:25.338185 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.338194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.338252 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.338425 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.338426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.338427 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0923 22:17:25.338481 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:25.338483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.339459 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.339460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:25.340064 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:25.340066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.340553 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.340554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:25.340612 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.340642 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 22:17:25.353662 (+ 13020us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 22:17:25.353769 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.129.126:44893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.129.126:44893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 22:17:25.354552 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.338031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.1.129.126:44893)
0923 22:17:25.338103 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.338105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.338258 (+ 153us) server_negotiation.cc:407] Connection header received
0923 22:17:25.338326 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.338327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.338340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.338426 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0923 22:17:25.338540 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.339369 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.340146 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.340446 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.340520 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.340767 (+ 247us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 22:17:25.353507 (+ 12740us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 22:17:25.354371 (+ 864us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260923 22:17:25.354622 6736 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260923 22:17:25.354612 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 6665
2026-09-23T22:17:25Z chronyd exiting
I20260923 22:17:25.390569 1541 test_util.cc:175] -----------------------------------------------
I20260923 22:17:25.390713 1541 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201819212015-1541-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 22:17:25.391453 1541 mini_oidc.cc:152] Starting JWKS server
I20260923 22:17:25.392130 1541 webserver.cc:460] Webserver started at https://127.0.0.1:44157/ using document root <none> and password file <none>
I20260923 22:17:25.392198 1541 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 22:17:25.392369 1541 webserver.cc:460] Webserver started at http://127.0.0.1:37699/ using document root <none> and password file <none>
2026-09-23T22:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:25Z Disabled control of system clock
I20260923 22:17:25.399876 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:33019
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:33019
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44157/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/testchainca.cert with env {}
W20260923 22:17:25.478550 6768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:25.478780 6768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:25.478801 6768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:25.479988 6768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:25.480029 6768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:25.480042 6768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:25.480055 6768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:25.480067 6768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 22:17:25.480079 6768 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 22:17:25.480091 6768 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44157/jwks.json
I20260923 22:17:25.481581 6768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:33019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:33019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44157/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:25.481805 6768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:25.482004 6768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:25.486681 6773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.486740 6768 server_base.cc:1034] running on GCE node
W20260923 22:17:25.486668 6774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:25.486685 6776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.487089 6768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:25.487383 6768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:25.488528 6768 hybrid_clock.cc:648] HybridClock initialized: now 1790201845488507 us; error 36 us; skew 500 ppm
I20260923 22:17:25.489753 6768 webserver.cc:460] Webserver started at http://127.1.129.126:40383/ using document root <none> and password file <none>
I20260923 22:17:25.490007 6768 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:25.490056 6768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:25.490217 6768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:25.491303 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "d5e5af4fb0e04233a791a8c94e837954"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "73b081ecef773f55c98c3f6263ed2e51"
server_key_iv: "d1834ea233c55081d5cf57c3b9495445"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.491660 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "d5e5af4fb0e04233a791a8c94e837954"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "73b081ecef773f55c98c3f6263ed2e51"
server_key_iv: "d1834ea233c55081d5cf57c3b9495445"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.492940 6768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:25.493654 6782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:25.493825 6768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:25.493914 6768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "d5e5af4fb0e04233a791a8c94e837954"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "73b081ecef773f55c98c3f6263ed2e51"
server_key_iv: "d1834ea233c55081d5cf57c3b9495445"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.493988 6768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:25.524716 6768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:25.525048 6768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:25.525203 6768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:25.529022 6768 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:33019
I20260923 22:17:25.529078 6834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:33019 every 8 connection(s)
I20260923 22:17:25.529430 6768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:25.530395 6835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:25.532629 6835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954: Bootstrap starting.
I20260923 22:17:25.533200 6835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:25.533478 6835 log.cc:824] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:25.534046 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 6768
I20260923 22:17:25.534157 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:25.534260 6835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954: No bootstrap required, opened a new log
I20260923 22:17:25.534332 1541 external_mini_cluster.cc:1428] Setting key 599aabc6c55d157fe3a6154849c7047b
I20260923 22:17:25.536598 6835 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e5af4fb0e04233a791a8c94e837954" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33019 } }
I20260923 22:17:25.536734 6835 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:25.536759 6835 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e5af4fb0e04233a791a8c94e837954, State: Initialized, Role: FOLLOWER
I20260923 22:17:25.536893 6835 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e5af4fb0e04233a791a8c94e837954" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33019 } }
I20260923 22:17:25.536972 6835 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:25.537014 6835 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:25.537063 6835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:25.537616 6835 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e5af4fb0e04233a791a8c94e837954" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33019 } }
I20260923 22:17:25.537730 6835 leader_election.cc:302] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5e5af4fb0e04233a791a8c94e837954; no voters:
I20260923 22:17:25.537880 6835 leader_election.cc:288] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:25.537899 6840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:25.538004 6840 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [term 1 LEADER]: Becoming Leader. State: Replica: d5e5af4fb0e04233a791a8c94e837954, State: Running, Role: LEADER
I20260923 22:17:25.538130 6835 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:25.538137 6840 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e5af4fb0e04233a791a8c94e837954" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33019 } }
I20260923 22:17:25.538528 6840 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5e5af4fb0e04233a791a8c94e837954. Latest consensus state: current_term: 1 leader_uuid: "d5e5af4fb0e04233a791a8c94e837954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e5af4fb0e04233a791a8c94e837954" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33019 } } }
I20260923 22:17:25.538781 6840 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:25.538703 6841 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5e5af4fb0e04233a791a8c94e837954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e5af4fb0e04233a791a8c94e837954" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33019 } } }
I20260923 22:17:25.538959 6841 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:25.539906 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.534830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.1.129.126:33019)
0923 22:17:25.535060 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.535069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.535101 (+ 32us) server_negotiation.cc:407] Connection header received
0923 22:17:25.535544 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.535550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.535606 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.535749 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.535970 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.536880 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.537731 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.538326 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.538774 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.539208 (+ 434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.539212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.539212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.539406 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.539438 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.539448 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.539497 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.539614 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.539706 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
W20260923 22:17:25.540387 6855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:25.540490 6855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:25.540596 6856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:25.540757 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:25.540786 6856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:25.542305 6856 catalog_manager.cc:1347] Generated new cluster ID: e34f48678c2e45899c0f4dbd9700847e
I20260923 22:17:25.542366 6856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:25.555814 6856 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:25.556309 6856 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:25.558976 6856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5e5af4fb0e04233a791a8c94e837954: Generated new TSK 0
I20260923 22:17:25.559154 6856 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 22:17:25.582597 6865 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 22:17:25.582706 6865 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 22:17:25.583078 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.568991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.1.129.126:33019)
0923 22:17:25.569035 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.569037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.569115 (+ 78us) server_negotiation.cc:407] Connection header received
0923 22:17:25.569204 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.569205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.569224 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.569303 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0923 22:17:25.569453 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.570003 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.570872 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.571079 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.571148 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.571351 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 22:17:25.582837 (+ 11486us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 22:17:25.583006 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:17:25.583140 6839 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 22:17:25.583148 6864 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:25.568923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:33019 (local address 127.0.0.1:44636)
0923 22:17:25.569066 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.569075 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.569133 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.569301 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.569303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.569305 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 22:17:25.569372 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:25.569374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.570100 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.570101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:25.570773 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:25.570775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.571163 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.571164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:25.571219 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.571224 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 22:17:25.582952 (+ 11728us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 22:17:25.583048 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.129.126:33019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260923 22:17:25.583520 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 6768
2026-09-23T22:17:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 22:17:25.621200 1541 mini_oidc.cc:152] Starting JWKS server
I20260923 22:17:25.621935 1541 webserver.cc:460] Webserver started at https://127.0.0.1:38607/ using document root <none> and password file <none>
I20260923 22:17:25.621979 1541 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 22:17:25.622154 1541 webserver.cc:460] Webserver started at http://127.0.0.1:46359/ using document root <none> and password file <none>
2026-09-23T22:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:25Z Disabled control of system clock
I20260923 22:17:25.629742 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35905
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:35905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38607/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 22:17:25.708760 6871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:25.708998 6871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:25.709034 6871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:25.710206 6871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:25.710273 6871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:25.710295 6871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:25.710318 6871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:25.710356 6871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 22:17:25.710382 6871 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 22:17:25.710409 6871 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38607/jwks.json
I20260923 22:17:25.711879 6871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:35905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38607/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:25.712172 6871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:25.712476 6871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:25.717603 6877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:25.717612 6876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.717667 6871 server_base.cc:1034] running on GCE node
W20260923 22:17:25.717603 6879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.718030 6871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:25.718354 6871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:25.719520 6871 hybrid_clock.cc:648] HybridClock initialized: now 1790201845719498 us; error 35 us; skew 500 ppm
I20260923 22:17:25.720728 6871 webserver.cc:460] Webserver started at http://127.1.129.126:35541/ using document root <none> and password file <none>
I20260923 22:17:25.721011 6871 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:25.721082 6871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:25.721233 6871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:25.722328 6871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "a50b995c871341429162b309f1dad9eb"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "9fe75705973832e7da7929ca53a06b70"
server_key_iv: "a40b42c64b11b43634160580b53a9f25"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.722779 6871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "a50b995c871341429162b309f1dad9eb"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "9fe75705973832e7da7929ca53a06b70"
server_key_iv: "a40b42c64b11b43634160580b53a9f25"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.724159 6871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:25.724887 6885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:25.725070 6871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:25.725170 6871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "a50b995c871341429162b309f1dad9eb"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "9fe75705973832e7da7929ca53a06b70"
server_key_iv: "a40b42c64b11b43634160580b53a9f25"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.725262 6871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:25.773815 6871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:25.774210 6871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:25.774374 6871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:25.778154 6871 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:35905
I20260923 22:17:25.778191 6937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:35905 every 8 connection(s)
I20260923 22:17:25.778566 6871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:25.779266 6938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:25.781885 6938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb: Bootstrap starting.
I20260923 22:17:25.782466 6938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:25.782816 6938 log.cc:824] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:25.783512 6938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb: No bootstrap required, opened a new log
I20260923 22:17:25.784475 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 6871
I20260923 22:17:25.784557 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:25.784742 1541 external_mini_cluster.cc:1428] Setting key b5cd7d2fbd1218cdf05303e0798a415a
I20260923 22:17:25.785272 6938 raft_consensus.cc:348] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50b995c871341429162b309f1dad9eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35905 } }
I20260923 22:17:25.785485 6938 raft_consensus.cc:374] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:25.785552 6938 raft_consensus.cc:729] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50b995c871341429162b309f1dad9eb, State: Initialized, Role: FOLLOWER
I20260923 22:17:25.785686 6938 consensus_queue.cc:260] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50b995c871341429162b309f1dad9eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35905 } }
I20260923 22:17:25.785750 6938 raft_consensus.cc:388] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:25.785771 6938 raft_consensus.cc:482] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:25.785836 6938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:25.786396 6938 raft_consensus.cc:504] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50b995c871341429162b309f1dad9eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35905 } }
I20260923 22:17:25.786478 6938 leader_election.cc:302] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a50b995c871341429162b309f1dad9eb; no voters:
I20260923 22:17:25.786689 6938 leader_election.cc:288] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:25.786882 6943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:25.786983 6938 sys_catalog.cc:564] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:25.787048 6943 raft_consensus.cc:686] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [term 1 LEADER]: Becoming Leader. State: Replica: a50b995c871341429162b309f1dad9eb, State: Running, Role: LEADER
I20260923 22:17:25.787163 6943 consensus_queue.cc:237] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50b995c871341429162b309f1dad9eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35905 } }
I20260923 22:17:25.787590 6944 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a50b995c871341429162b309f1dad9eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50b995c871341429162b309f1dad9eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35905 } } }
I20260923 22:17:25.787676 6944 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:25.787830 6945 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a50b995c871341429162b309f1dad9eb. Latest consensus state: current_term: 1 leader_uuid: "a50b995c871341429162b309f1dad9eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50b995c871341429162b309f1dad9eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35905 } } }
I20260923 22:17:25.787879 6945 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:25.788218 6952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:25.788789 6952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:25.790336 6952 catalog_manager.cc:1347] Generated new cluster ID: 2e32e98b27b9499d843ae06e59ff0d37
I20260923 22:17:25.790385 6952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:25.790874 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.785315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.1.129.126:35905)
0923 22:17:25.785573 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.785583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.785617 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:17:25.785733 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.785763 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.785836 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.785964 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.786178 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.787407 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.788599 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.789181 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.789305 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.790163 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.790167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.790167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.790370 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.790410 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.790424 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.790490 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.790573 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.790669 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
I20260923 22:17:25.791720 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:25.799976 6952 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:25.800472 6952 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:25.802677 6952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a50b995c871341429162b309f1dad9eb: Generated new TSK 0
I20260923 22:17:25.802860 6952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:25.812155 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.808089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.1.129.126:35905)
0923 22:17:25.808152 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.808154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.808259 (+ 105us) server_negotiation.cc:407] Connection header received
0923 22:17:25.808345 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.808346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.808356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.808414 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0923 22:17:25.808562 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.809019 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.810168 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.810367 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.810417 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.810610 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 22:17:25.811946 (+ 1336us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 22:17:25.812079 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 22:17:25.812208 6942 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 22:17:25.812256 6967 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:25.808023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35905 (local address 127.0.0.1:60308)
0923 22:17:25.808200 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:25.808212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:25.808277 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:25.808431 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:25.808432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:25.808433 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0923 22:17:25.808499 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:25.808501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.809106 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.809107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:25.810083 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:25.810086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.810449 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.810451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:25.810517 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.810523 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 22:17:25.812053 (+ 1530us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 22:17:25.812156 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.129.126:35905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260923 22:17:25.814132 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 6871
E20260923 22:17:25.822980 6968 webserver.cc:533] Webserver: sslize failed:
E20260923 22:17:25.823060 6968 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T22:17:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T22:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:25Z Disabled control of system clock
I20260923 22:17:25.860119 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:46529
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:46529
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/testchainca.cert with env {}
W20260923 22:17:25.938946 6972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:25.939137 6972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:25.939160 6972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:25.940229 6972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:25.940264 6972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:25.940276 6972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:25.940289 6972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:25.940301 6972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 22:17:25.940313 6972 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 22:17:25.940325 6972 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 22:17:25.941709 6972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:46529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:25.941890 6972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:25.942034 6972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:25.946841 6978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:25.946862 6980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.946874 6972 server_base.cc:1034] running on GCE node
W20260923 22:17:25.946830 6977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:25.947227 6972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:25.947559 6972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:25.948743 6972 hybrid_clock.cc:648] HybridClock initialized: now 1790201845948725 us; error 41 us; skew 500 ppm
I20260923 22:17:25.949950 6972 webserver.cc:460] Webserver started at http://127.1.129.126:44173/ using document root <none> and password file <none>
I20260923 22:17:25.950202 6972 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:25.950300 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:25.950438 6972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:25.951510 6972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "b8b5c0e6f0304249873e1593700b2487"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "557ec0c5d0b3dc2e3fcca433a51de6cd"
server_key_iv: "52e968ddab92b05a893ffae787d6831f"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.951864 6972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "b8b5c0e6f0304249873e1593700b2487"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "557ec0c5d0b3dc2e3fcca433a51de6cd"
server_key_iv: "52e968ddab92b05a893ffae787d6831f"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.953119 6972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:25.953810 6986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:25.954006 6972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:25.954103 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "b8b5c0e6f0304249873e1593700b2487"
format_stamp: "Formatted at 2026-09-23 22:17:25 on dist-test-slave-dmw7"
server_key: "557ec0c5d0b3dc2e3fcca433a51de6cd"
server_key_iv: "52e968ddab92b05a893ffae787d6831f"
server_key_version: "encryptionkey@0"
I20260923 22:17:25.954174 6972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:25.972534 6972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:25.972898 6972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:25.973059 6972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:25.976953 6972 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:46529
I20260923 22:17:25.977010 7038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:46529 every 8 connection(s)
I20260923 22:17:25.977393 6972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:25.978369 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:25.980791 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487: Bootstrap starting.
I20260923 22:17:25.981426 7039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:25.981722 7039 log.cc:824] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:25.982456 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487: No bootstrap required, opened a new log
I20260923 22:17:25.984084 7039 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5c0e6f0304249873e1593700b2487" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46529 } }
I20260923 22:17:25.984236 7039 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:25.984289 7039 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b5c0e6f0304249873e1593700b2487, State: Initialized, Role: FOLLOWER
I20260923 22:17:25.984467 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 6972
I20260923 22:17:25.984452 7039 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5c0e6f0304249873e1593700b2487" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46529 } }
I20260923 22:17:25.984535 7039 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:25.984555 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:25.984570 7039 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:25.984629 7039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:25.984771 1541 external_mini_cluster.cc:1428] Setting key 7f54eaeffa99f60415e68e198f37cce7
I20260923 22:17:25.985311 7039 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5c0e6f0304249873e1593700b2487" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46529 } }
I20260923 22:17:25.985460 7039 leader_election.cc:302] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b5c0e6f0304249873e1593700b2487; no voters:
I20260923 22:17:25.985634 7039 leader_election.cc:288] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:25.985687 7044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:25.985868 7044 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [term 1 LEADER]: Becoming Leader. State: Replica: b8b5c0e6f0304249873e1593700b2487, State: Running, Role: LEADER
I20260923 22:17:25.985888 7039 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:25.985996 7044 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5c0e6f0304249873e1593700b2487" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46529 } }
I20260923 22:17:25.986316 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b5c0e6f0304249873e1593700b2487. Latest consensus state: current_term: 1 leader_uuid: "b8b5c0e6f0304249873e1593700b2487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5c0e6f0304249873e1593700b2487" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46529 } } }
I20260923 22:17:25.986406 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:25.986320 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8b5c0e6f0304249873e1593700b2487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5c0e6f0304249873e1593700b2487" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46529 } } }
I20260923 22:17:25.986493 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:25.986943 7052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:25.987664 7052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:25.989308 7052 catalog_manager.cc:1347] Generated new cluster ID: 5478a0eddd4a4bd4b107cec2adafa747
I20260923 22:17:25.989355 7052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:25.991482 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:25.985333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.1.129.126:46529)
0923 22:17:25.985560 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:25.985570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:25.985607 (+ 37us) server_negotiation.cc:407] Connection header received
0923 22:17:25.985739 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:25.985747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:25.985827 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:25.986277 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:25.986386 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.988177 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.989151 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:25.989748 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:25.989858 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:25.990618 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:25.990622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:25.990622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:25.990855 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:25.990908 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:25.990924 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:25.990989 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:25.991181 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 22:17:25.991273 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260923 22:17:25.992626 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:26.003732 7052 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:26.004475 7052 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:26.008949 7052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b5c0e6f0304249873e1593700b2487: Generated new TSK 0
I20260923 22:17:26.009155 7052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:26.054075 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.014043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.1.129.126:46529)
0923 22:17:26.014110 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:26.014112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:26.014265 (+ 153us) server_negotiation.cc:407] Connection header received
0923 22:17:26.014335 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:26.014336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:26.014347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:26.014405 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0923 22:17:26.014579 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.015172 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.016322 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.016535 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.016587 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.016776 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 22:17:26.053868 (+ 37092us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 22:17:26.054017 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56776: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 22:17:26.054164 7043 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56776: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 22:17:26.054286 7068 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:26.013984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46529 (local address 127.0.0.1:56776)
0923 22:17:26.014163 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:26.014175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:26.014257 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:26.014420 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:26.014421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:26.014423 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 22:17:26.014495 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:26.014497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.015264 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.015265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:26.016226 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:26.016229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.016612 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.016613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:26.016675 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.016683 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 22:17:26.054023 (+ 37340us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 22:17:26.054171 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.129.126:46529: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":116,"threads_started":1}
I20260923 22:17:26.054622 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 6972
2026-09-23T22:17:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (220 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:26 dist-test-slave-dmw7 krb5kdc[7080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:26 dist-test-slave-dmw7 krb5kdc[7080](info): set up 2 sockets
Sep 23 22:17:26 dist-test-slave-dmw7 krb5kdc[7080](info): commencing operation
krb5kdc: starting...
W20260923 22:17:27.258900 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:27 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:27.378609 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:45645
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:45645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:27.458551 7093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:27.458866 7093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:27.458905 7093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:27.460124 7093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:27.460206 7093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:27.460232 7093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:27.460253 7093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:27.460269 7093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:27.461771 7093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:45645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:27.462060 7093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:27.462302 7093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:27.467331 7099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:27.467309 7101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:27.467542 7098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:27.467625 7093 server_base.cc:1034] running on GCE node
I20260923 22:17:27.467906 7093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:27.468267 7093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:27.469488 7093 hybrid_clock.cc:648] HybridClock initialized: now 1790201847469463 us; error 40 us; skew 500 ppm
Sep 23 22:17:27 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:27.472185 7093 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:27.472709 7093 webserver.cc:460] Webserver started at http://127.1.129.126:36859/ using document root <none> and password file <none>
I20260923 22:17:27.472950 7093 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:27.473019 7093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:27.473179 7093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:27.474275 7093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "36416d70195d436e83c2dfaf9ec30b3a"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "0daae48f37c6d7b5c71fb904ef857576"
server_key_iv: "609d32f484e2c12ddb8e286eaef3e3e4"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.474730 7093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "36416d70195d436e83c2dfaf9ec30b3a"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "0daae48f37c6d7b5c71fb904ef857576"
server_key_iv: "609d32f484e2c12ddb8e286eaef3e3e4"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.476147 7093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:27.476987 7108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:27.477192 7093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:27.477300 7093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "36416d70195d436e83c2dfaf9ec30b3a"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "0daae48f37c6d7b5c71fb904ef857576"
server_key_iv: "609d32f484e2c12ddb8e286eaef3e3e4"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.477392 7093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:27.506031 7093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:27.508069 7093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:27.508252 7093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:27.512389 7093 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:45645
I20260923 22:17:27.512436 7160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:45645 every 8 connection(s)
I20260923 22:17:27.512825 7093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:27.513674 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:27.513957 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 7093
I20260923 22:17:27.514060 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:27.514250 1541 external_mini_cluster.cc:1428] Setting key 2780cea51decfd9fed35932ec5af5f5c
I20260923 22:17:27.517032 7161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a: Bootstrap starting.
I20260923 22:17:27.517759 7161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:27.518060 7161 log.cc:824] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:27.518899 7161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a: No bootstrap required, opened a new log
Sep 23 22:17:27 dist-test-slave-dmw7 krb5kdc[7080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:27.521142 7161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36416d70195d436e83c2dfaf9ec30b3a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45645 } }
I20260923 22:17:27.521315 7161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:27.521353 7161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36416d70195d436e83c2dfaf9ec30b3a, State: Initialized, Role: FOLLOWER
I20260923 22:17:27.521497 7161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36416d70195d436e83c2dfaf9ec30b3a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45645 } }
I20260923 22:17:27.521586 7161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:27.521623 7161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:27.521693 7161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:27.522411 7161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36416d70195d436e83c2dfaf9ec30b3a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45645 } }
I20260923 22:17:27.522533 7161 leader_election.cc:302] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36416d70195d436e83c2dfaf9ec30b3a; no voters:
I20260923 22:17:27.522750 7161 leader_election.cc:288] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:27.522814 7166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:27.522961 7166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [term 1 LEADER]: Becoming Leader. State: Replica: 36416d70195d436e83c2dfaf9ec30b3a, State: Running, Role: LEADER
I20260923 22:17:27.523063 7161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:27.523119 7166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36416d70195d436e83c2dfaf9ec30b3a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45645 } }
I20260923 22:17:27.523509 7166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36416d70195d436e83c2dfaf9ec30b3a. Latest consensus state: current_term: 1 leader_uuid: "36416d70195d436e83c2dfaf9ec30b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36416d70195d436e83c2dfaf9ec30b3a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45645 } } }
I20260923 22:17:27.523521 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36416d70195d436e83c2dfaf9ec30b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36416d70195d436e83c2dfaf9ec30b3a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 45645 } } }
I20260923 22:17:27.523671 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:27.523869 7166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:27.524039 7173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:27.524595 7173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:27.525694 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:27.514825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.1.129.126:45645)
0923 22:17:27.515001 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:27.515009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:27.515034 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:17:27.515453 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:27.515459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:27.515505 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:27.515621 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:27.516394 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.517322 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.518404 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.518958 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.519081 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:27.521759 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:27.521763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:27.521763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:27.521773 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:27.523853 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:27.524278 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:27.524280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:27.524281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:27.524387 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:27.524617 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:27.524619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:27.524619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:27.524968 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:27.525114 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:27.525339 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 22:17:27.525447 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260923 22:17:27.527906 7173 catalog_manager.cc:1347] Generated new cluster ID: 60dd2da0e1a0458698231fd79f3411ff
I20260923 22:17:27.527963 7173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:27.546993 7173 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:27.547565 7173 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:27.555706 7173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36416d70195d436e83c2dfaf9ec30b3a: Generated new TSK 0
I20260923 22:17:27.555919 7173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:27.583427 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45645
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:27.665436 7187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:27.665655 7187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:27.665680 7187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:27.666910 7187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:27.666958 7187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:27.666986 7187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:27.668596 7187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:27.668867 7187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:27.669114 7187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:27.674402 7195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:27.674557 7192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:27.674610 7187 server_base.cc:1034] running on GCE node
W20260923 22:17:27.674402 7193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:27.674942 7187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:27.675189 7187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:27.676395 7187 hybrid_clock.cc:648] HybridClock initialized: now 1790201847676385 us; error 34 us; skew 500 ppm
Sep 23 22:17:27 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:27.679083 7187 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:27.679576 7187 webserver.cc:460] Webserver started at http://127.1.129.65:40345/ using document root <none> and password file <none>
I20260923 22:17:27.679807 7187 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:27.679886 7187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:27.680037 7187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:27.681224 7187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "1592f4cc6a9b440485aade054c89d32c"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "07e1566d9b6e522f9d45da285176890e"
server_key_iv: "8cd15bf3d519c5ef70c8355a25ac3754"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.681605 7187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "1592f4cc6a9b440485aade054c89d32c"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "07e1566d9b6e522f9d45da285176890e"
server_key_iv: "8cd15bf3d519c5ef70c8355a25ac3754"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.682992 7187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:27.683795 7202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:27.683976 7187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:27.684082 7187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "1592f4cc6a9b440485aade054c89d32c"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "07e1566d9b6e522f9d45da285176890e"
server_key_iv: "8cd15bf3d519c5ef70c8355a25ac3754"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.684159 7187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:27.699755 7187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:27.701650 7187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:27.701834 7187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:27.702085 7187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:27.702410 7187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:27.702468 7187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:27.702509 7187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:27.702538 7187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:27.708623 7187 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:45357
I20260923 22:17:27.708673 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:45357 every 8 connection(s)
I20260923 22:17:27.709043 7187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:27 dist-test-slave-dmw7 krb5kdc[7080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:27.718077 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:27.709850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:47109 (local address 127.1.129.126:45645)
0923 22:17:27.709926 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:27.709928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:27.710123 (+ 195us) server_negotiation.cc:407] Connection header received
0923 22:17:27.711079 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:27.711080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:27.711090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:27.711151 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:27.711748 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.712237 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.713187 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.713381 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.713432 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:27.715511 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:27.715513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:27.715514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:27.715521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:27.716932 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:27.717298 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:27.717299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:27.717300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:27.717352 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:27.717561 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:27.717561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:27.717561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:27.717662 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:27.717710 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:27.717944 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 22:17:27.718010 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 22:17:27.718310 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:27.709836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45645 (local address 127.1.129.65:47109)
0923 22:17:27.709994 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:27.710037 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:27.710988 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:27.711206 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:27.711209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:27.711473 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:27.711668 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:27.711672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.712323 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.712324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:27.713105 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:27.713107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.713462 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.713462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:27.713569 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:27.714044 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:27.714052 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:27.715410 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:27.717008 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:27.717014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:27.717016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:27.717226 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:27.717409 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:27.717410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:27.717410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:27.717508 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:27.717714 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:27.717718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:27.717835 (+ 117us) client_negotiation.cc:769] Sending connection context
0923 22:17:27.717918 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 22:17:27.718037 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260923 22:17:27.718366 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 7187
I20260923 22:17:27.718446 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:27.718593 1541 external_mini_cluster.cc:1428] Setting key 2dcb7c47b1447805b76ff0027b5ca324
I20260923 22:17:27.719231 7316 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45645
I20260923 22:17:27.719345 7316 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:27.719547 7316 heartbeater.cc:511] Master 127.1.129.126:45645 requested a full tablet report, sending...
I20260923 22:17:27.720031 7125 ts_manager.cc:194] Registered new tserver with Master: 1592f4cc6a9b440485aade054c89d32c (127.1.129.65:45357)
I20260923 22:17:27.720650 7125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:47109
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:27.745090 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45645
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:27.827179 7321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:27.827386 7321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:27.827411 7321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:27.828498 7321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:27.828534 7321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:27.828560 7321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:27.829943 7321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:27.830120 7321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:27.830292 7321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:27.835129 7327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:27.835129 7326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:27.835388 7321 server_base.cc:1034] running on GCE node
W20260923 22:17:27.835170 7329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:27.835677 7321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:27.835939 7321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:27.837100 7321 hybrid_clock.cc:648] HybridClock initialized: now 1790201847837089 us; error 24 us; skew 500 ppm
Sep 23 22:17:27 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:27.839668 7321 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:27.840029 7321 webserver.cc:460] Webserver started at http://127.1.129.66:43073/ using document root <none> and password file <none>
I20260923 22:17:27.840200 7321 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:27.840238 7321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:27.840330 7321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:27.841270 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "9019d688444146738fe4ccdcb3e4e1b7"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "742f1ef82e7e2151738a8bc5885d5e2b"
server_key_iv: "63d0ffccbf81d00ddce49c50054adb4d"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.841569 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "9019d688444146738fe4ccdcb3e4e1b7"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "742f1ef82e7e2151738a8bc5885d5e2b"
server_key_iv: "63d0ffccbf81d00ddce49c50054adb4d"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.842833 7321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:27.843544 7336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:27.843737 7321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:27.843838 7321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "9019d688444146738fe4ccdcb3e4e1b7"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-dmw7"
server_key: "742f1ef82e7e2151738a8bc5885d5e2b"
server_key_iv: "63d0ffccbf81d00ddce49c50054adb4d"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.843890 7321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:27.877841 7321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:27.879805 7321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:27.879954 7321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:27.880208 7321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:27.880525 7321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:27.880580 7321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:27.880625 7321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:27.880654 7321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:27.886597 7321 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:36013
I20260923 22:17:27.886706 7449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:36013 every 8 connection(s)
I20260923 22:17:27.887097 7321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:27.890476 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 7321
I20260923 22:17:27.890580 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:27.890796 1541 external_mini_cluster.cc:1428] Setting key 5e0534d204540b7b59a0a1efa2777401
Sep 23 22:17:27 dist-test-slave-dmw7 krb5kdc[7080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:27.898303 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:27.887970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:48775 (local address 127.1.129.126:45645)
0923 22:17:27.888016 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:27.888018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:27.888265 (+ 247us) server_negotiation.cc:407] Connection header received
0923 22:17:27.889256 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:27.889257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:27.889270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:27.889318 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:27.889984 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.890642 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.891932 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.892252 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.892329 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:27.894922 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:27.894924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:27.894924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:27.894933 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:27.896703 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:27.897135 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:27.897136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:27.897136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:27.897205 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:27.897509 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:27.897511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:27.897511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:27.897671 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:27.897746 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:27.898075 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 22:17:27.898177 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:27.898387 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:27.887981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45645 (local address 127.1.129.66:48775)
0923 22:17:27.888142 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:27.888179 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:27.889134 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:27.889402 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:27.889405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:27.889670 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:27.889862 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:27.889866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.890765 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.890766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:27.891840 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:27.891842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.892373 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.892374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:27.892489 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:27.893087 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:27.893096 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:27.894820 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:27.896799 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:27.896806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:27.896808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:27.897056 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:27.897301 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:27.897302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:27.897302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:27.897431 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:27.897780 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:27.897785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:27.897925 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 22:17:27.898024 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 22:17:27.898168 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260923 22:17:27.899170 7450 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45645
I20260923 22:17:27.899302 7450 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:27.899500 7450 heartbeater.cc:511] Master 127.1.129.126:45645 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
I20260923 22:17:27.899941 7125 ts_manager.cc:194] Registered new tserver with Master: 9019d688444146738fe4ccdcb3e4e1b7 (127.1.129.66:36013)
I20260923 22:17:27.900322 7125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:48775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:27.917665 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45645
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:27.997603 7455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:27.997844 7455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:27.997882 7455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:27.999464 7455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:27.999712 7455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:27.999795 7455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:28.002005 7455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:28.002326 7455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:28.002569 7455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:28.007560 7460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:28.007604 7461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:28.007774 7455 server_base.cc:1034] running on GCE node
W20260923 22:17:28.007588 7463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:28.008064 7455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:28.008272 7455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:28.009465 7455 hybrid_clock.cc:648] HybridClock initialized: now 1790201848009440 us; error 42 us; skew 500 ppm
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.012038 7455 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:28.012545 7455 webserver.cc:460] Webserver started at http://127.1.129.67:41409/ using document root <none> and password file <none>
I20260923 22:17:28.012789 7455 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:28.012858 7455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:28.013008 7455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:28.014060 7455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "e02cc75c21c34a3ca493d9d398ca94cc"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-dmw7"
server_key: "9dd552ae2b86ac2e6aee8e30c8bb93f7"
server_key_iv: "4b1cd44887615e350d0ac5a5d7bd9733"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.014420 7455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "e02cc75c21c34a3ca493d9d398ca94cc"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-dmw7"
server_key: "9dd552ae2b86ac2e6aee8e30c8bb93f7"
server_key_iv: "4b1cd44887615e350d0ac5a5d7bd9733"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.015829 7455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:28.016570 7470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.016747 7455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:28.016844 7455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "e02cc75c21c34a3ca493d9d398ca94cc"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-dmw7"
server_key: "9dd552ae2b86ac2e6aee8e30c8bb93f7"
server_key_iv: "4b1cd44887615e350d0ac5a5d7bd9733"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.016925 7455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:28.030982 7455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:28.032920 7455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:28.033084 7455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:28.033314 7455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:28.033633 7455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:28.033696 7455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.033766 7455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:28.033795 7455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.039661 7455 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:33057
I20260923 22:17:28.039701 7583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:33057 every 8 connection(s)
I20260923 22:17:28.040022 7455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:28.042421 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 7455
I20260923 22:17:28.042510 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:28.042693 1541 external_mini_cluster.cc:1428] Setting key b7ff788401ac860440c4a41ae291b9dd
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:28.049642 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.040805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:49225 (local address 127.1.129.126:45645)
0923 22:17:28.040873 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.040875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.041054 (+ 179us) server_negotiation.cc:407] Connection header received
0923 22:17:28.042032 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.042033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.042043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.042100 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.042975 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.043671 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.044637 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.044830 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.044885 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.047109 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.047110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.047110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.047117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.048588 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.048930 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.048931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.048932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.048979 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.049175 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.049175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.049175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.049274 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.049321 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.049523 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.049578 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:17:28.049840 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.040782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:45645 (local address 127.1.129.67:49225)
0923 22:17:28.040951 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:28.040987 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:28.041917 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:28.042174 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:28.042177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:28.042578 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:28.042906 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:28.042911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.043747 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.043748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:28.044553 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:28.044555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.044901 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.044902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:28.045005 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.045496 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:28.045503 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:28.046978 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:28.048651 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:28.048656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:28.048658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:28.048868 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:28.049032 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:28.049032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:28.049033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:28.049125 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:28.049324 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:28.049327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:28.049435 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 22:17:28.049514 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 22:17:28.049624 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260923 22:17:28.050359 7584 heartbeater.cc:347] Connected to a master server at 127.1.129.126:45645
I20260923 22:17:28.050464 7584 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:28.050669 7584 heartbeater.cc:511] Master 127.1.129.126:45645 requested a full tablet report, sending...
I20260923 22:17:28.051041 7125 ts_manager.cc:194] Registered new tserver with Master: e02cc75c21c34a3ca493d9d398ca94cc (127.1.129.67:33057)
I20260923 22:17:28.051396 7125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:49225
I20260923 22:17:28.054409 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/test-admin.keytab.
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.071247 1541 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 22:17:28.072993 7591 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:28.072251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.1.129.65:45357)
0923 22:17:28.072411 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.072424 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.072435 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:28.072461 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.072463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.072477 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.072549 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.072887 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59814: BlockingRecv error: recv got EOF from 127.0.0.1:59814 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260923 22:17:28.073170 1541 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5cc' of type FILE
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.074194 1541 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.075179 1541 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:28.083465 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.075730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.1.129.65:45357)
0923 22:17:28.075784 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.075786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.075880 (+ 94us) server_negotiation.cc:407] Connection header received
0923 22:17:28.075960 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.075961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.075973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.076045 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.076448 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.077169 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.077939 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.078292 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.078375 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.080681 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.080692 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.080692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.080726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.082189 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.082587 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.082591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.082591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.082691 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.082897 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.082898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.082898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.083047 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.083202 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.083262 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.083343 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260923 22:17:28.085348 7591 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:28.084697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.1.129.65:45357)
0923 22:17:28.084732 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.084734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.084836 (+ 102us) server_negotiation.cc:407] Connection header received
0923 22:17:28.084908 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.084909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.084920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.084974 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.085298 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59832: BlockingRecv error: recv got EOF from 127.0.0.1:59832 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260923 22:17:28.085417 1541 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5cc' of type FILE
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.086418 1541 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.087375 1541 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:28.094102 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.087871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.1.129.65:45357)
0923 22:17:28.087922 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.087924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.087981 (+ 57us) server_negotiation.cc:407] Connection header received
0923 22:17:28.088053 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.088054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.088063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.088105 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.088470 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.088968 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.089661 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.089874 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.089957 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.091858 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.091860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.091860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.091868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.093002 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.093362 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.093363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.093364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.093420 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.093613 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.093614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.093614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.093710 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.093818 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.093922 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.094001 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260923 22:17:28.095592 7591 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:28.095053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.1.129.65:45357)
0923 22:17:28.095113 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.095115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.095149 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:17:28.095219 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.095220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.095228 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.095282 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.095543 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59856: BlockingRecv error: recv got EOF from 127.0.0.1:59856 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 22:17:28.095657 1541 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201819212015-1541-0/krb5kdc/krb5cc' of type FILE
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.096553 1541 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.097357 1541 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:17:28.104096 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.097802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.1.129.65:45357)
0923 22:17:28.097866 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.097867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.097927 (+ 60us) server_negotiation.cc:407] Connection header received
0923 22:17:28.098018 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.098019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.098027 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.098093 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.098447 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.098984 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.099653 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.099862 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.099936 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.101832 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.101833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.101833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.101840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.103035 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.103377 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.103378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.103378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.103431 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.103621 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.103622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.103622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.103717 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.103811 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.103926 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.103997 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:17:28.104701 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 7187
I20260923 22:17:28.109844 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 7321
I20260923 22:17:28.114089 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 7455
I20260923 22:17:28.118185 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 7093
2026-09-23T22:17:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1891 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7604](info): set up 2 sockets
Sep 23 22:17:28 dist-test-slave-dmw7 krb5kdc[7604](info): commencing operation
krb5kdc: starting...
W20260923 22:17:29.138962 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.126.keytab.
Entry for principal oryx/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.126.keytab.
I20260923 22:17:29.255738 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37359
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.126.keytab
--principal=oryx/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:29.335259 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:29.335469 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:29.335492 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:29.336650 7617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:29.336694 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:29.336709 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:29.336721 7617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:29.336733 7617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:29.338310 7617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.126.keytab
--principal=oryx/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:29.338570 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:29.338819 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:29.343616 7617 server_base.cc:1034] running on GCE node
W20260923 22:17:29.343621 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.343598 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.343600 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:29.344099 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:29.344437 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:29.345630 7617 hybrid_clock.cc:648] HybridClock initialized: now 1790201849345610 us; error 30 us; skew 500 ppm
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:29.358541 7617 init.cc:377] Logged in from keytab as oryx/127.1.129.126@KRBTEST.COM (short username oryx)
I20260923 22:17:29.359033 7617 webserver.cc:460] Webserver started at http://127.1.129.126:39997/ using document root <none> and password file <none>
I20260923 22:17:29.359269 7617 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:29.359339 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:29.359495 7617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:29.360567 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "6ff3279a277842738477b74a18bf8336"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "5686d8672f89ef73e2f45cdaf2c2dfa2"
server_key_iv: "a3e29016c98c22a6ed0d169ee2ef34fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.360925 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "6ff3279a277842738477b74a18bf8336"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "5686d8672f89ef73e2f45cdaf2c2dfa2"
server_key_iv: "a3e29016c98c22a6ed0d169ee2ef34fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.362222 7617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:29.363008 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.363193 7617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.363294 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "6ff3279a277842738477b74a18bf8336"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "5686d8672f89ef73e2f45cdaf2c2dfa2"
server_key_iv: "a3e29016c98c22a6ed0d169ee2ef34fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.363376 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:29.376093 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:29.377898 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:29.378054 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:29.381822 7617 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:37359
I20260923 22:17:29.381878 7684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:37359 every 8 connection(s)
I20260923 22:17:29.382208 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:29.383177 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:29.385376 7685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: Bootstrap starting.
I20260923 22:17:29.386004 7685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:29.386302 7685 log.cc:824] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:29.386987 7685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: No bootstrap required, opened a new log
I20260923 22:17:29.388469 7685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } }
I20260923 22:17:29.388597 7685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:29.388617 7685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff3279a277842738477b74a18bf8336, State: Initialized, Role: FOLLOWER
I20260923 22:17:29.388799 7685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } }
I20260923 22:17:29.388864 7685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:29.388885 7685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:29.388948 7685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:29.389719 7685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } }
I20260923 22:17:29.389799 7685 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff3279a277842738477b74a18bf8336; no voters:
I20260923 22:17:29.390013 7685 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:29.390053 7688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:29.390187 7688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff3279a277842738477b74a18bf8336, State: Running, Role: LEADER
I20260923 22:17:29.390257 7685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:29.390388 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 7617
I20260923 22:17:29.390389 7688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } }
I20260923 22:17:29.390481 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:29.390698 1541 external_mini_cluster.cc:1428] Setting key 7cacf24d05a3c559c8de76f0d8e8f588
I20260923 22:17:29.390786 7689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } }
I20260923 22:17:29.390849 7689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:29.390990 7690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff3279a277842738477b74a18bf8336. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } }
I20260923 22:17:29.391044 7690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:29.391587 7695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:29.392064 7695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:29.393527 7695 catalog_manager.cc:1347] Generated new cluster ID: 9437cbf915884997a3a22450e6fd2837
I20260923 22:17:29.393577 7695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:29.401908 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:29.391337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.1.129.126:37359)
0923 22:17:29.391549 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:29.391558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:29.391586 (+ 28us) server_negotiation.cc:407] Connection header received
0923 22:17:29.391733 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:29.391741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:29.391817 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:29.391948 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:29.392685 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.393975 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.395006 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.395632 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.395764 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:29.398413 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:29.398416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:29.398417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:29.398428 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:29.400302 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.400706 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.400708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.400709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.400824 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.401029 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.401030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.401030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.401319 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:29.401458 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:29.401585 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 22:17:29.401720 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260923 22:17:29.406932 7695 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:29.407595 7695 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:29.410482 7695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: Generated new TSK 0
I20260923 22:17:29.410671 7695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab.
Entry for principal oryx/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab.
I20260923 22:17:29.437758 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37359
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab
--principal=oryx/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:29.516770 7711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:29.516983 7711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:29.517005 7711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:29.518200 7711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:29.518240 7711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:29.518266 7711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:29.519711 7711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab
--principal=oryx/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:29.519941 7711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:29.520162 7711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:29.525041 7717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:29.525082 7711 server_base.cc:1034] running on GCE node
W20260923 22:17:29.525132 7719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.525236 7716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:29.525549 7711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:29.525744 7711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:29.526899 7711 hybrid_clock.cc:648] HybridClock initialized: now 1790201849526877 us; error 47 us; skew 500 ppm
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:29.529373 7711 init.cc:377] Logged in from keytab as oryx/127.1.129.65@KRBTEST.COM (short username oryx)
I20260923 22:17:29.529781 7711 webserver.cc:460] Webserver started at http://127.1.129.65:46439/ using document root <none> and password file <none>
I20260923 22:17:29.530015 7711 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:29.530085 7711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:29.530236 7711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:29.531312 7711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "d1f50f21c8f849eeb01f9ed22733ac91"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "c586c127198e45c12fe1e9beb7df1014"
server_key_iv: "910fc1bdf7f1012ae6f33714cca03f09"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.531674 7711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "d1f50f21c8f849eeb01f9ed22733ac91"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "c586c127198e45c12fe1e9beb7df1014"
server_key_iv: "910fc1bdf7f1012ae6f33714cca03f09"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.532943 7711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:29.533670 7726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.533841 7711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:29.533938 7711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "d1f50f21c8f849eeb01f9ed22733ac91"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "c586c127198e45c12fe1e9beb7df1014"
server_key_iv: "910fc1bdf7f1012ae6f33714cca03f09"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.534013 7711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:29.550586 7711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:29.552404 7711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:29.552565 7711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:29.552783 7711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:29.553089 7711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:29.553143 7711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.553185 7711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:29.553212 7711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.558780 7711 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:32841
I20260923 22:17:29.558858 7839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:32841 every 8 connection(s)
I20260923 22:17:29.559172 7711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:29.562273 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 7711
I20260923 22:17:29.562388 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:29.562585 1541 external_mini_cluster.cc:1428] Setting key efaceb0d33a46feb05cbc3949df53a3e
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:29.569387 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:29.559943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:59771 (local address 127.1.129.126:37359)
0923 22:17:29.559995 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:29.559998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:29.560173 (+ 175us) server_negotiation.cc:407] Connection header received
0923 22:17:29.560673 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:29.560674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:29.560686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:29.560743 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:29.561651 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.562216 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.563563 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.563889 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.563959 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:29.566224 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:29.566226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:29.566226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:29.566233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:29.568026 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.568423 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.568425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.568425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.568488 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.568725 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.568726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.568727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.568880 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:29.568960 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:29.569188 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 22:17:29.569275 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:17:29.569406 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:29.559871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.65:59771)
0923 22:17:29.560026 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:29.560067 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:29.560568 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:29.560818 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:29.560821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:29.561370 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:29.561551 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:29.561556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.562332 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.562333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:29.563478 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:29.563480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.563968 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.563969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:29.564066 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:29.564609 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:29.564616 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:29.566138 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:29.568125 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:29.568131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:29.568133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:29.568347 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:29.568558 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:29.568559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:29.568559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:29.568655 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:29.568950 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:29.568954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:29.569059 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 22:17:29.569137 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 22:17:29.569251 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260923 22:17:29.569936 7840 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37359
I20260923 22:17:29.570039 7840 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:29.570209 7840 heartbeater.cc:511] Master 127.1.129.126:37359 requested a full tablet report, sending...
I20260923 22:17:29.570780 7649 ts_manager.cc:194] Registered new tserver with Master: d1f50f21c8f849eeb01f9ed22733ac91 (127.1.129.65:32841)
WARNING: no policy specified for oryx/127.1.129.66@KRBTEST.COM; defaulting to no policy
I20260923 22:17:29.571710 7649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:59771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab.
Entry for principal oryx/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab.
I20260923 22:17:29.589112 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37359
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab
--principal=oryx/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:29.668329 7845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:29.668515 7845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:29.668538 7845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:29.669658 7845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:29.669701 7845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:29.669727 7845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:29.671239 7845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab
--principal=oryx/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:29.671485 7845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:29.671710 7845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:29.676692 7850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.676697 7853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.676702 7851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:29.677037 7845 server_base.cc:1034] running on GCE node
I20260923 22:17:29.677245 7845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:29.677435 7845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:29.678571 7845 hybrid_clock.cc:648] HybridClock initialized: now 1790201849678561 us; error 27 us; skew 500 ppm
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:29.680930 7845 init.cc:377] Logged in from keytab as oryx/127.1.129.66@KRBTEST.COM (short username oryx)
I20260923 22:17:29.681383 7845 webserver.cc:460] Webserver started at http://127.1.129.66:34993/ using document root <none> and password file <none>
I20260923 22:17:29.681617 7845 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:29.681661 7845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:29.681820 7845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:29.682925 7845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "3d7571de4fee408e94e2d7001258bd55"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "b5362735e02de3f1a5910f805f030e6e"
server_key_iv: "289e104f559d899195f98c72702b45ea"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.683344 7845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "3d7571de4fee408e94e2d7001258bd55"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "b5362735e02de3f1a5910f805f030e6e"
server_key_iv: "289e104f559d899195f98c72702b45ea"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.684612 7845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:29.685351 7860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.685549 7845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:29.685637 7845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "3d7571de4fee408e94e2d7001258bd55"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "b5362735e02de3f1a5910f805f030e6e"
server_key_iv: "289e104f559d899195f98c72702b45ea"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.685709 7845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:29.694407 7845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:29.696609 7845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:29.696784 7845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:29.697037 7845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:29.697403 7845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:29.697474 7845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.697512 7845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:29.697537 7845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.705214 7845 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:39403
I20260923 22:17:29.705261 7973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:39403 every 8 connection(s)
I20260923 22:17:29.705572 7845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:29.713729 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 7845
I20260923 22:17:29.713825 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:29.714015 1541 external_mini_cluster.cc:1428] Setting key 9f1c0d1fca07c9db8fbb25aa75292444
I20260923 22:17:29.714293 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:29.706270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:41163 (local address 127.1.129.126:37359)
0923 22:17:29.706340 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:29.706342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:29.706509 (+ 167us) server_negotiation.cc:407] Connection header received
0923 22:17:29.707042 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:29.707044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:29.707054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:29.707110 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:29.707972 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.708487 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.709407 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.709599 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.709655 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:29.711636 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:29.711637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:29.711637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:29.711643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:29.712999 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.713319 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.713320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.713320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.713370 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.713572 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.713572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.713572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.713669 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:29.713720 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:29.714074 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 22:17:29.714181 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:17:29.714419 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:29.706227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:41163)
0923 22:17:29.706385 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:29.706426 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:29.706958 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:29.707160 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:29.707163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:29.707693 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:29.707896 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:29.707901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.708562 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.708563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:29.709320 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:29.709322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.709671 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.709672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:29.709766 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:29.710207 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:29.710215 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:29.711553 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:29.713066 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:29.713070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:29.713071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:29.713260 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:29.713426 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:29.713426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:29.713426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:29.713514 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:29.713740 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:29.713745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:29.713916 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 22:17:29.714035 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 22:17:29.714196 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260923 22:17:29.714848 7974 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37359
I20260923 22:17:29.714958 7974 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:29.715126 7974 heartbeater.cc:511] Master 127.1.129.126:37359 requested a full tablet report, sending...
I20260923 22:17:29.715528 7649 ts_manager.cc:194] Registered new tserver with Master: 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403)
I20260923 22:17:29.716029 7649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:41163
WARNING: no policy specified for oryx/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab.
Entry for principal oryx/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab.
I20260923 22:17:29.737936 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37359
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab
--principal=oryx/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:29.818624 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:29.818912 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:29.818950 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:29.820108 7979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:29.820190 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:29.820236 7979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:29.821687 7979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab
--principal=oryx/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:29.821964 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:29.822234 7979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:29.827142 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.827261 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.827162 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:29.827390 7979 server_base.cc:1034] running on GCE node
I20260923 22:17:29.827545 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:29.827713 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:29.828845 7979 hybrid_clock.cc:648] HybridClock initialized: now 1790201849828826 us; error 38 us; skew 500 ppm
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:29.831368 7979 init.cc:377] Logged in from keytab as oryx/127.1.129.67@KRBTEST.COM (short username oryx)
I20260923 22:17:29.831784 7979 webserver.cc:460] Webserver started at http://127.1.129.67:36815/ using document root <none> and password file <none>
I20260923 22:17:29.832023 7979 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:29.832091 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:29.832242 7979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:29.833279 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "c7c59dfe33c347d690d5f0ce962522f6"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "0eb66be4dfb77337a02352e1461d5757"
server_key_iv: "b1e20f010ba95d3e6de2ef6db13ca7a5"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.833652 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "c7c59dfe33c347d690d5f0ce962522f6"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "0eb66be4dfb77337a02352e1461d5757"
server_key_iv: "b1e20f010ba95d3e6de2ef6db13ca7a5"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.834981 7979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:29.835716 7994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.835920 7979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:29.836016 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "c7c59dfe33c347d690d5f0ce962522f6"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "0eb66be4dfb77337a02352e1461d5757"
server_key_iv: "b1e20f010ba95d3e6de2ef6db13ca7a5"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.836097 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:29.848048 7979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:29.851311 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:29.851465 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:29.851703 7979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:29.852049 7979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:29.852097 7979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.852131 7979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:29.852171 7979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.858561 7979 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:36269
I20260923 22:17:29.858608 8107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:36269 every 8 connection(s)
I20260923 22:17:29.858956 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:29.862982 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 7979
I20260923 22:17:29.863054 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:29.863262 1541 external_mini_cluster.cc:1428] Setting key 249c41cef59d591d8a0978cb6c377d7d
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:29.868232 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:29.859786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:55001 (local address 127.1.129.126:37359)
0923 22:17:29.859869 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:29.859871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:29.860064 (+ 193us) server_negotiation.cc:407] Connection header received
0923 22:17:29.860567 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:29.860568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:29.860579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:29.860640 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:29.861636 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.862125 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.863275 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.863531 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.863618 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:29.865773 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:29.865774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:29.865774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:29.865780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:29.867155 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.867492 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.867493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.867493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.867542 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.867749 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.867750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.867750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.867848 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:29.867897 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:29.868114 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 22:17:29.868168 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 22:17:29.868450 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:29.859732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:55001)
0923 22:17:29.859910 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:29.859976 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:29.860458 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:29.860763 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:29.860767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:29.861361 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:29.861555 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:29.861560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.862215 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.862215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:29.863198 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:29.863201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.863622 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.863623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:29.863765 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:29.864288 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:29.864296 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:29.865699 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:29.867223 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:29.867228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:29.867230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:29.867429 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:29.867597 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:29.867597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:29.867598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:29.867691 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:29.867900 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:29.867904 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:29.868007 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 22:17:29.868087 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:17:29.868204 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260923 22:17:29.868922 8108 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37359
I20260923 22:17:29.869004 8108 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:29.869228 8108 heartbeater.cc:511] Master 127.1.129.126:37359 requested a full tablet report, sending...
I20260923 22:17:29.869593 7649 ts_manager.cc:194] Registered new tserver with Master: c7c59dfe33c347d690d5f0ce962522f6 (127.1.129.67:36269)
I20260923 22:17:29.869963 7649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:55001
I20260923 22:17:29.874899 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.129.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.125.keytab.
Entry for principal oryx/127.1.129.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.125.keytab.
I20260923 22:17:29.901074 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.125:45633
--webserver_interface=127.1.129.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.125.keytab
--principal=oryx/127.1.129.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:37359,127.1.129.125:45633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:29.981359 8113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:29.981562 8113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:29.981583 8113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:29.982580 8113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:29.982612 8113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:29.982625 8113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:29.982712 8113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:29.982724 8113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:29.984067 8113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:37359,127.1.129.125:45633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.125.keytab
--principal=oryx/127.1.129.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.125:45633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.129.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
W20260923 22:17:29.984156 8113 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.129.126:37359,127.1.129.125:45633'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 22:17:29.984261 8113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:29.984411 8113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:29.989055 8119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.989074 8118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.989022 8121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:29.989425 8113 server_base.cc:1034] running on GCE node
I20260923 22:17:29.989642 8113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:29.989934 8113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:29.991075 8113 hybrid_clock.cc:648] HybridClock initialized: now 1790201849991063 us; error 28 us; skew 500 ppm
Sep 23 22:17:29 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:29.993505 8113 init.cc:377] Logged in from keytab as oryx/127.1.129.125@KRBTEST.COM (short username oryx)
I20260923 22:17:29.993952 8113 webserver.cc:460] Webserver started at http://127.1.129.125:34523/ using document root <none> and password file <none>
I20260923 22:17:29.994170 8113 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:29.994212 8113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:29.994364 8113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:29.995457 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data/instance:
uuid: "3a0956b8965348e587ec5485f0ce8064"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "5e4a65335ed724515c7e81a4e561a565"
server_key_iv: "c4422106265d32a5e2c7b0c06639b86d"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.995836 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal/instance:
uuid: "3a0956b8965348e587ec5485f0ce8064"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "5e4a65335ed724515c7e81a4e561a565"
server_key_iv: "c4422106265d32a5e2c7b0c06639b86d"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.997085 8113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:29.997803 8128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.997995 8113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:29.998080 8113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
uuid: "3a0956b8965348e587ec5485f0ce8064"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "5e4a65335ed724515c7e81a4e561a565"
server_key_iv: "c4422106265d32a5e2c7b0c06639b86d"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.998147 8113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.023309 8113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.025506 8113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.025648 8113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.029793 8113 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.125:45633
I20260923 22:17:30.029827 8180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.125:45633 every 8 connection(s)
I20260923 22:17:30.030236 8113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
I20260923 22:17:30.031318 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.032902 8181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:17:30.036100 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 8113
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.125@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.129.125@KRBTEST.COM
I20260923 22:17:30.042850 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.033615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.1.129.126:37359)
0923 22:17:30.033702 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.033704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.033892 (+ 188us) server_negotiation.cc:407] Connection header received
0923 22:17:30.034312 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.034312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.034322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.034382 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.035154 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.035640 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.037011 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.037332 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.037410 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.039766 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.039768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.039768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.039775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.041297 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.041700 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.041701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.041702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.041762 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.042041 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.042043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.042043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.042250 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.042320 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.042670 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.042758 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:17:30.043083 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.033571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37140)
0923 22:17:30.033768 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.033809 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.034232 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.034452 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.034455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.034836 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.035038 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.035043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.035727 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.035728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.036929 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.036931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.037463 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.037464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.037568 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.038104 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.038114 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.039643 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.041394 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.041401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.041403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.041624 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.041862 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.041862 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.041863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.041969 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.042319 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.042323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.042499 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.042604 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.042824 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260923 22:17:30.043800 8181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:17:30.045116 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.036682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.1.129.125:45633)
0923 22:17:30.036852 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.036856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.036876 (+ 20us) server_negotiation.cc:407] Connection header received
0923 22:17:30.036897 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.036899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.036917 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.036995 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.037667 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.038804 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.039621 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.039990 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.040069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.042240 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.042245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.042245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.042250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.043919 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.044298 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.044302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.044302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.044368 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.044576 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.044577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.044577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.044756 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.044886 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.044993 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.045050 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.125@KRBTEST.COM for oryx/127.1.129.125@KRBTEST.COM
I20260923 22:17:30.050357 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.044109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.1.129.125:45633)
0923 22:17:30.044263 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.044265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.044273 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:30.044342 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.044343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.044359 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.044442 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.044717 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.045443 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.046105 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.046299 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.046354 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.047906 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.047907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.047907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.047912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.049513 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.049794 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.049795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.049795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.049842 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.049997 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.049997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.049997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.050077 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.050121 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.050216 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.050275 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260923 22:17:30.050395 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.044059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.0.0.1:39938)
0923 22:17:30.044225 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.044233 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.044286 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.044447 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.044447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.044591 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.044643 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.044645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.045509 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.045509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.046042 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.046044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.046368 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.046369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.046417 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.046784 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.046788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.047839 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.049577 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.049578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.049578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.049733 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.049907 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.049907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.049908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.049954 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.050124 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.050125 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.050155 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.050203 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.050317 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":46}
I20260923 22:17:30.051676 8181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Bootstrap starting.
I20260923 22:17:30.052287 8181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.052546 8181 log.cc:824] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.053205 8181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: No bootstrap required, opened a new log
I20260923 22:17:30.054546 8181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } }
I20260923 22:17:30.054718 8181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.054769 8181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0956b8965348e587ec5485f0ce8064, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.054896 8181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } }
I20260923 22:17:30.055190 8181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:30.055162 8188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } } }
I20260923 22:17:30.055263 8188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:17:30.056766 8199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:30.056821 8199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:30.058357 8113 master_runner.cc:428] Detected that this master 3a0956b8965348e587ec5485f0ce8064 is joining an existing cluster
I20260923 22:17:30.058408 8113 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.129.125:45633
I20260923 22:17:30.066192 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.059623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.1.129.126:37359)
0923 22:17:30.059675 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.059677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.059698 (+ 21us) server_negotiation.cc:407] Connection header received
0923 22:17:30.059722 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.059723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.059734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.059799 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.060304 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.060798 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.061554 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.061770 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.061862 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.063148 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.063150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.063150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.063157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.064953 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.065391 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.065393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.065393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.065457 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.065629 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.065629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.065630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.065772 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.065848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.066024 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.066102 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:17:30.066286 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.059506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.1.129.125:45633)
0923 22:17:30.059549 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.059551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.059752 (+ 201us) server_negotiation.cc:407] Connection header received
0923 22:17:30.059839 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.059839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.059847 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.059873 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.060325 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.061051 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.062067 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.062319 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.062404 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.063390 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.063391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.063391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.063398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.065046 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.065474 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.065475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.065475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.065527 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.065759 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.065760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.065760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.065879 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.065939 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.066094 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.066198 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":25}
I20260923 22:17:30.066457 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.059424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37150)
0923 22:17:30.059570 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.059581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.059645 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.059803 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.059804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.060115 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.060210 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.060212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.060894 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.060895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.061450 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.061452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.061916 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.061917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.061974 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.062477 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.062482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.063055 (+ 573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.065048 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.065050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.065050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.065299 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.065515 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.065516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.065516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.065577 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.065840 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.065841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.065882 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.065984 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.066149 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":15,"thread_start_us":89,"threads_started":1}
I20260923 22:17:30.066901 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.059489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.0.0.1:39950)
0923 22:17:30.059683 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.059693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.059773 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.059933 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.059933 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.060170 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.060242 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.060244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.061138 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.061138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.061983 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.061985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.062450 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.062451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.062510 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.062913 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.062918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.063294 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.065134 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.065135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.065135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.065402 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.065583 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.065584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.065584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.065651 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.065945 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.065946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.065981 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.066121 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.066786 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.126@KRBTEST.COM for oryx/127.1.129.125@KRBTEST.COM
I20260923 22:17:30.076233 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.068785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.0.0.1:39962)
0923 22:17:30.068950 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.068992 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.069069 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.069246 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.069250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.069604 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.069668 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.069670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.070323 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.070324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.071279 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.071282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.071613 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.071615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.071706 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.072106 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.072111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.073603 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.075151 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.075157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.075160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.075356 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.075556 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.075556 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.075557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.075620 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.075869 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.075876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.075926 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.076026 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.076137 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260923 22:17:30.076232 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.068869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.1.129.125:45633)
0923 22:17:30.068910 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.068912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.069051 (+ 139us) server_negotiation.cc:407] Connection header received
0923 22:17:30.069168 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.069169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.069178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.069228 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.069783 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.070244 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.071373 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.071543 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.071597 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.073716 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.073717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.073718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.073724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.075055 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.075432 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.075433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.075433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.075490 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.075700 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.075701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.075701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.075802 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.075876 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.076063 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.076135 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:30.076747 7649 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.129.125:45633: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3a0956b8965348e587ec5485f0ce8064"
member_type: NON_VOTER
last_known_addr {
host: "127.1.129.125"
port: 45633
}
attrs {
promote: true
}
}
dest_uuid: "6ff3279a277842738477b74a18bf8336"
cas_config_opid_index: -1
I20260923 22:17:30.077047 7649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } }
I20260923 22:17:30.077493 8209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3a0956b8965348e587ec5485f0ce8064 (127.1.129.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } }
I20260923 22:17:30.077832 8155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.078606 8209 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.129.125:45633
I20260923 22:17:30.078814 8155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Refusing update from remote peer 6ff3279a277842738477b74a18bf8336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 22:17:30.078840 8113 master.cc:537] Master@127.1.129.125:45633 shutting down...
I20260923 22:17:30.079396 8209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260923 22:17:30.078843 8208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } }
I20260923 22:17:30.081274 8208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:30.081537 7634 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260923 22:17:30.081745 8113 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:30.081830 8113 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:30.081885 8113 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: stopping tablet replica
W20260923 22:17:30.081923 7634 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 -> Peer 3a0956b8965348e587ec5485f0ce8064 (127.1.129.125:45633): Couldn't send request to peer 3a0956b8965348e587ec5485f0ce8064. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 22:17:30.094460 8113 master.cc:552] Master@127.1.129.125:45633 shutdown complete.
I20260923 22:17:30.094511 8113 master_runner.cc:305] Clearing existing system tablet
I20260923 22:17:30.094681 8113 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.094730 8113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.095353 8113 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:30.095860 8213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.095994 8113 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.096050 8113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.096079 8113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
uuid: "3a0956b8965348e587ec5485f0ce8064"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "5e4a65335ed724515c7e81a4e561a565"
server_key_iv: "c4422106265d32a5e2c7b0c06639b86d"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.096779 8113 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:30.098047 8113 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 22:17:30.098130 8113 log.cc:1196] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 22:17:30.098323 8113 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Deleting consensus metadata
I20260923 22:17:30.098461 8113 master_runner.cc:315] Copying system tablet from 127.1.129.126:37359
I20260923 22:17:30.099141 8113 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: tablet copy: Beginning tablet copy session from remote peer at address 127.1.129.126:37359
I20260923 22:17:30.105432 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.099524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.1.129.126:37359)
0923 22:17:30.099575 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.099577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.099647 (+ 70us) server_negotiation.cc:407] Connection header received
0923 22:17:30.099712 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.099713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.099722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.099766 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.100117 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.100597 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.101642 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.101815 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.101870 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.103003 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.103005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.103005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.103011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.104387 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.104759 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.104760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.104760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.104809 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.105001 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.105001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.105001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.105098 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.105145 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.105307 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.105367 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:17:30.105494 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.099437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37154)
0923 22:17:30.099565 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.099577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.099649 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.099781 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.099782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.099986 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.100052 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.100054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.100673 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.100674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.101554 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.101556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.101887 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.101888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.101955 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.102415 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.102421 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.102924 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.104465 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.104467 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.104467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.104685 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.104876 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.104877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.104877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.104939 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.105153 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.105154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.105199 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.105283 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.105386 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":83,"threads_started":1}
I20260923 22:17:30.105799 7669 tablet_copy_service.cc:140] P 6ff3279a277842738477b74a18bf8336: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3a0956b8965348e587ec5485f0ce8064 ({username='oryx', principal='oryx/127.1.129.125@KRBTEST.COM'} at 127.0.0.1:37154)
I20260923 22:17:30.105901 7669 tablet_copy_service.cc:161] P 6ff3279a277842738477b74a18bf8336: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3a0956b8965348e587ec5485f0ce8064 at {username='oryx', principal='oryx/127.1.129.125@KRBTEST.COM'} at 127.0.0.1:37154: session id = 3a0956b8965348e587ec5485f0ce8064-00000000000000000000000000000000
I20260923 22:17:30.106801 7669 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 22:17:30.107383 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 22:17:30.108301 8113 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: tablet copy: Starting download of 0 data blocks...
I20260923 22:17:30.108462 8113 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: tablet copy: Starting download of 1 WAL segments...
I20260923 22:17:30.109314 8113 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 22:17:30.110126 7669 tablet_copy_service.cc:342] P 6ff3279a277842738477b74a18bf8336: Request end of tablet copy session 3a0956b8965348e587ec5485f0ce8064-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.129.125@KRBTEST.COM'} at 127.0.0.1:37154
I20260923 22:17:30.110184 7669 tablet_copy_service.cc:433] P 6ff3279a277842738477b74a18bf8336: ending tablet copy session 3a0956b8965348e587ec5485f0ce8064-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3a0956b8965348e587ec5485f0ce8064
I20260923 22:17:30.112497 8113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.113814 8221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.113879 8220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.114058 8223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.114091 8113 server_base.cc:1034] running on GCE node
I20260923 22:17:30.114243 8113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.114389 8113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.115513 8113 hybrid_clock.cc:648] HybridClock initialized: now 1790201850115506 us; error 25 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.116796 8113 init.cc:377] Logged in from keytab as oryx/127.1.129.125@KRBTEST.COM (short username oryx)
I20260923 22:17:30.117105 8113 webserver.cc:460] Webserver started at http://127.1.129.125:36159/ using document root <none> and password file <none>
I20260923 22:17:30.117240 8113 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.117280 8113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.117893 8113 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.118429 8230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.118549 8113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.118595 8113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
uuid: "3a0956b8965348e587ec5485f0ce8064"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "5e4a65335ed724515c7e81a4e561a565"
server_key_iv: "c4422106265d32a5e2c7b0c06639b86d"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.118676 8113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.137584 8113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.139360 8113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.142540 8113 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.125:45633
I20260923 22:17:30.142757 8282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.125:45633 every 8 connection(s)
I20260923 22:17:30.142940 8113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
I20260923 22:17:30.143636 8283 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:17:30.144042 8283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Bootstrap starting.
I20260923 22:17:30.146190 8283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:17:30.146427 8283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Bootstrap complete.
I20260923 22:17:30.146790 8283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } }
I20260923 22:17:30.146883 8283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a0956b8965348e587ec5485f0ce8064, State: Initialized, Role: LEARNER
I20260923 22:17:30.146924 8283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } }
I20260923 22:17:30.147092 8287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } }
I20260923 22:17:30.147178 8287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:30.147104 8283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:30.148816 8298 catalog_manager.cc:1259] Loaded cluster ID: 9437cbf915884997a3a22450e6fd2837
I20260923 22:17:30.148865 8298 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: loading cluster ID for follower catalog manager: success
I20260923 22:17:30.149952 8298 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: acquiring CA information for follower catalog manager: success
I20260923 22:17:30.150600 8298 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:17:30.153713 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.146617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.1.129.125:45633)
0923 22:17:30.146767 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.146769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.146778 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.146802 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.146803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.146814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.146897 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.147561 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.148676 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.149470 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.149791 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.149863 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.151154 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.151156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.151156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.151163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.152578 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.152926 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.152928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.152928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.153002 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.153202 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.153203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.153204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.153317 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.153420 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.153528 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.153616 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":127,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for oryx/127.1.129.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.124.keytab.
Entry for principal oryx/127.1.129.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.124.keytab.
I20260923 22:17:30.182950 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.124:33931
--webserver_interface=127.1.129.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.124.keytab
--principal=oryx/127.1.129.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:37359,127.1.129.125:45633,127.1.129.124:33931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:30.264925 8301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.265147 8301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.265173 8301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.266474 8301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:30.266517 8301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.266531 8301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.266572 8301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:30.266597 8301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:30.268132 8301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:37359,127.1.129.125:45633,127.1.129.124:33931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.124.keytab
--principal=oryx/127.1.129.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.124:33931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.129.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:30.268399 8301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.268638 8301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.273525 8309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.273501 8307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.273505 8306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.273855 8301 server_base.cc:1034] running on GCE node
I20260923 22:17:30.274089 8301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.274325 8301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.275465 8301 hybrid_clock.cc:648] HybridClock initialized: now 1790201850275450 us; error 28 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.278141 8301 init.cc:377] Logged in from keytab as oryx/127.1.129.124@KRBTEST.COM (short username oryx)
I20260923 22:17:30.278584 8301 webserver.cc:460] Webserver started at http://127.1.129.124:45309/ using document root <none> and password file <none>
I20260923 22:17:30.278854 8301 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.278903 8301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.279067 8301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:30.280117 8301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data/instance:
uuid: "a4d323ca0bf345db9234c194b536f76e"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-dmw7"
server_key: "10c42e61bd3821bae43db222a5c69a2b"
server_key_iv: "a5cd738e55d4a1185702aa03646815d8"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.280476 8301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal/instance:
uuid: "a4d323ca0bf345db9234c194b536f76e"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-dmw7"
server_key: "10c42e61bd3821bae43db222a5c69a2b"
server_key_iv: "a5cd738e55d4a1185702aa03646815d8"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.281996 8301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:30.282780 8316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.282968 8301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.283039 8301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
uuid: "a4d323ca0bf345db9234c194b536f76e"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-dmw7"
server_key: "10c42e61bd3821bae43db222a5c69a2b"
server_key_iv: "a5cd738e55d4a1185702aa03646815d8"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.283126 8301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.302481 8301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.304852 8301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.305008 8301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.309475 8301 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.124:33931
I20260923 22:17:30.309509 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.124:33931 every 8 connection(s)
I20260923 22:17:30.309873 8301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
I20260923 22:17:30.311007 8369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.312481 8369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.124@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:30.318186 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 8301
I20260923 22:17:30.321777 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.313267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.1.129.126:37359)
0923 22:17:30.313342 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.313344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.313440 (+ 96us) server_negotiation.cc:407] Connection header received
0923 22:17:30.313883 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.313884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.313893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.313952 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.314903 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.315387 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.316359 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.316533 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.316582 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.318799 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.318800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.318801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.318809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.320387 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.320833 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.320834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.320834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.320885 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.321146 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.321147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.321147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.321287 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.321357 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.321596 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.321674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:17:30.321872 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.313124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37156)
0923 22:17:30.313310 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.313352 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.313800 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.314034 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.314038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.314519 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.314818 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.314824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.315467 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.315468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.316277 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.316279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.316611 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.316612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.316715 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.317184 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.317199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.318696 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.320465 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.320471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.320472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.320760 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.320950 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.320951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.320951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.321084 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.321357 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.321360 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.321472 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.321553 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.321673 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260923 22:17:30.322381 8369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.129.124@KRBTEST.COM
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.124@KRBTEST.COM for oryx/127.1.129.125@KRBTEST.COM
I20260923 22:17:30.327773 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.318855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.1.129.124:33931)
0923 22:17:30.319030 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.319057 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.319083 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:17:30.319105 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.319107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.319121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.319196 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.320026 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.320944 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.321900 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.322357 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.322424 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.324855 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.324863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.324864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.324870 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.326497 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.326910 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.326914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.326914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.326972 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.327183 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.327184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.327185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.327363 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.327529 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.327577 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.327654 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260923 22:17:30.329279 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.322698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.0.0.1:39990)
0923 22:17:30.322814 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.322824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.322894 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.323030 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.323031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.323164 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.323205 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.323206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.323862 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.323863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.324366 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.324367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.324854 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.324855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.324925 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.325292 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.325296 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.326667 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.328538 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.328539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.328539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.328690 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.328854 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.328854 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.328854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.328899 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.329074 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.329075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.329105 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.329152 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.329220 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260923 22:17:30.329293 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.322745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.1.129.125:45633)
0923 22:17:30.322812 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.322814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.322880 (+ 66us) server_negotiation.cc:407] Connection header received
0923 22:17:30.322945 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.322946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.322957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.323029 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.323279 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.323761 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.324452 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.324781 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.324848 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.326762 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.326763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.326763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.326770 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.328433 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.328748 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.328749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.328749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.328800 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.328936 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.328936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.328936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.329027 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.329072 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.329169 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.329234 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:30.329651 8369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.124@KRBTEST.COM for oryx/127.1.129.124@KRBTEST.COM
I20260923 22:17:30.335690 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.329903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.0.0.1:32790)
0923 22:17:30.329952 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.329963 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.329999 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.330098 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.330098 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.330202 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.330246 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.330247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.330876 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.330877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.331338 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.331339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.331644 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.331644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.331679 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.332074 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.332079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.333230 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.334731 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.334732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.334732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.334892 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.335088 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.335089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.335089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.335141 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.335386 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.335387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.335423 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.335489 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.335583 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:17:30.335736 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.329952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.1.129.124:33931)
0923 22:17:30.330010 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.330011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.330017 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:17:30.330034 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.330035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.330043 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.330093 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.330287 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.330806 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.331402 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.331585 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.331633 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.333323 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.333324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.333325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.333331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.334610 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.334971 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.334972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.334972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.335027 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.335211 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.335212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.335212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.335321 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.335394 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.335518 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.335631 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":41}
I20260923 22:17:30.337067 8369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Bootstrap starting.
I20260923 22:17:30.337965 8369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.338253 8369 log.cc:824] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.339094 8369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: No bootstrap required, opened a new log
I20260923 22:17:30.341133 8369 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } }
I20260923 22:17:30.341289 8369 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.341337 8369 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d323ca0bf345db9234c194b536f76e, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.341480 8369 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } }
I20260923 22:17:30.341791 8369 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:30.341761 8374 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } } }
I20260923 22:17:30.341885 8374 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:17:30.343711 8385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:30.343770 8385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:30.345996 8301 master_runner.cc:428] Detected that this master a4d323ca0bf345db9234c194b536f76e is joining an existing cluster
I20260923 22:17:30.346032 8301 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.129.124:33931
I20260923 22:17:30.354527 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.347144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.0.0.1:40004)
0923 22:17:30.347417 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.347428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.347507 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.347659 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.347659 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.347902 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.347986 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.347988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.348848 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.348849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.349795 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.349798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.350155 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.350156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.350223 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.350688 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.350694 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.351122 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.353304 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.353306 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.353306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.353537 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.353774 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.353775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.353775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.353842 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.354128 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.354130 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.354191 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.354271 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.354367 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":80,"threads_started":1}
I20260923 22:17:30.355226 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.347073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.1.129.126:37359)
0923 22:17:30.347181 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.347183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.347363 (+ 180us) server_negotiation.cc:407] Connection header received
0923 22:17:30.347468 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.347469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.347482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.347510 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.347964 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.348767 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.350878 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.351171 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.351232 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.352344 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.352345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.352346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.352353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.353933 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.354291 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.354292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.354292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.354354 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.354575 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.354576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.354577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.354747 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.354793 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.355023 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.355111 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 22:17:30.355261 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.347128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.1.129.125:45633)
0923 22:17:30.347231 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.347233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.347482 (+ 249us) server_negotiation.cc:407] Connection header received
0923 22:17:30.347583 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.347584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.347593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.347647 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.348060 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.348774 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.349873 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.350066 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.350143 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.351216 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.351218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.351218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.351226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.352980 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.353622 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.353624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.353624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.353698 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.353917 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.353918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.353919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.354054 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.354126 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.354461 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.354544 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 22:17:30.355572 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.347042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37158)
0923 22:17:30.347301 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.347313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.347389 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.347578 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.347578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.347796 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.347859 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.347861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.349915 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.349915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.350782 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.350784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.351288 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.351289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.351338 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.351880 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.351885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.352243 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.353997 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.353998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.353998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.354210 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.354427 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.354428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.354428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.354487 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.354841 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.354841 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.354886 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.355335 (+ 449us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.355460 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.126@KRBTEST.COM for oryx/127.1.129.124@KRBTEST.COM
I20260923 22:17:30.379379 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 7711
I20260923 22:17:30.384960 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:32841
--local_ip_for_outbound_sockets=127.1.129.65
--tserver_master_addrs=127.1.129.126:37359
--webserver_port=46439
--webserver_interface=127.1.129.65
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab
--principal=oryx/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633,127.1.129.124:33931 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:17:30.394325 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.357805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.0.0.1:32804)
0923 22:17:30.357859 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.357871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.357916 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.358183 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.358183 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.358357 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.358398 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.358399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.359383 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.359384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.359939 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.359940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.360413 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.360413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.360461 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.360855 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.360859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.362313 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.393193 (+ 30880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.393195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.393195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.393430 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.393639 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.393640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.393640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.393710 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.394004 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.394006 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.394059 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.394106 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.394222 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 22:17:30.394834 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.357859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.1.129.124:33931)
0923 22:17:30.358036 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.358038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.358046 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:30.358081 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.358081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.358094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.358163 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.358472 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.359293 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.360018 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.360340 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.360408 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.362431 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.362433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.362433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.362441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.393082 (+ 30641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.393513 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.393514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.393515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.393568 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.393779 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.393780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.393780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.393916 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.394507 (+ 591us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.394551 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.394671 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":108,"threads_started":1}
I20260923 22:17:30.395236 7649 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.129.124:33931: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a4d323ca0bf345db9234c194b536f76e"
member_type: NON_VOTER
last_known_addr {
host: "127.1.129.124"
port: 33931
}
attrs {
promote: true
}
}
dest_uuid: "6ff3279a277842738477b74a18bf8336"
cas_config_opid_index: 5
I20260923 22:17:30.395467 7649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } }
I20260923 22:17:30.395905 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.347427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.1.129.124:33931)
0923 22:17:30.347460 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.347462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.354714 (+ 7252us) server_negotiation.cc:407] Connection header received
0923 22:17:30.354800 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.354801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.354811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.354849 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.355776 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.356413 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.358235 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.358510 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.358552 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.391708 (+ 33156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.391711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.391711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.391720 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.393798 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.394214 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.394215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.394216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.394264 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.394795 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.394796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.394797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.394946 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.394991 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.395720 (+ 729us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.395793 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:30.396044 8343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.396203 8209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a4d323ca0bf345db9234c194b536f76e (127.1.129.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.396384 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.347271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.0.0.1:32800)
0923 22:17:30.354609 (+ 7338us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.354620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.354736 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.354900 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.354901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.355557 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.355612 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.355614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.356841 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.356842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.358134 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.358156 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.358602 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.358602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.358693 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.391189 (+ 32496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.391196 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.391590 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.393910 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.393911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.393912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.394133 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.394332 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.394332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.394332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.394710 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.395471 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.395472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.395518 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.396198 (+ 680us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.396291 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7327,"thread_start_us":79,"threads_started":1}
I20260923 22:17:30.397010 8209 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.129.124:33931
I20260923 22:17:30.397015 8343 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Refusing update from remote peer 6ff3279a277842738477b74a18bf8336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 22:17:30.397087 8209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.397177 8209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:30.397246 8301 master.cc:537] Master@127.1.129.124:33931 shutting down...
I20260923 22:17:30.397454 8209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:17:30.398205 7635 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 -> Peer a4d323ca0bf345db9234c194b536f76e (127.1.129.124:33931): Couldn't send request to peer a4d323ca0bf345db9234c194b536f76e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 22:17:30.399833 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.395887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.0.0.1:40018)
0923 22:17:30.395926 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.395936 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.395975 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.396187 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.396188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.396189 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.396238 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.396240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.397262 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.397263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.398564 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.398566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.399563 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.399564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.399637 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.399662 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.399731 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.399769 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 22:17:30.400036 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.395990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.1.129.125:45633)
0923 22:17:30.396033 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.396035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.396044 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.396073 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.396075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.396099 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.396155 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.396326 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.397049 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.398813 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.399468 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.399555 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.399635 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.399930 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.399972 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:30.400226 8257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 LEARNER]: Refusing update from remote peer 6ff3279a277842738477b74a18bf8336: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 22:17:30.400427 8209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 22:17:30.401012 8257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a4d323ca0bf345db9234c194b536f76e (127.1.129.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.401300 8301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:30.401377 8301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:30.401417 8301 tablet_replica.cc:330] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: stopping tablet replica
I20260923 22:17:30.402273 8287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff3279a277842738477b74a18bf8336. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.402350 8287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:30.402384 8287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: NON_VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: true } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.402423 8287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:30.402745 8209 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: attempting to promote NON_VOTER 3a0956b8965348e587ec5485f0ce8064 to VOTER
I20260923 22:17:30.402915 8209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } }
I20260923 22:17:30.403407 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.403260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.0.0.1:32812)
0923 22:17:30.403348 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.403380 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:33931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 22:17:30.403663 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.403593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.0.0.1:32826)
0923 22:17:30.403622 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.403642 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:33931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260923 22:17:30.403723 7635 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 -> Peer a4d323ca0bf345db9234c194b536f76e (127.1.129.124:33931): Couldn't send request to peer a4d323ca0bf345db9234c194b536f76e. Status: Network error: Client connection negotiation failed: client connection to 127.1.129.124:33931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 22:17:30.403981 8257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 LEARNER]: Refusing update from remote peer 6ff3279a277842738477b74a18bf8336: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 22:17:30.404158 8208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 22:17:30.404773 8208 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3a0956b8965348e587ec5485f0ce8064 (127.1.129.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.405610 8208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.405676 8208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:30.405736 8208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.405776 8208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:30.406078 8257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3a0956b8965348e587ec5485f0ce8064 (127.1.129.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.407027 8397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.407115 8397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:30.414721 8301 master.cc:552] Master@127.1.129.124:33931 shutdown complete.
I20260923 22:17:30.414774 8301 master_runner.cc:305] Clearing existing system tablet
I20260923 22:17:30.414906 8301 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.414949 8301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.415652 8301 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:30.416182 8400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.416335 8301 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.416393 8301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.416417 8301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
uuid: "a4d323ca0bf345db9234c194b536f76e"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-dmw7"
server_key: "10c42e61bd3821bae43db222a5c69a2b"
server_key_iv: "a5cd738e55d4a1185702aa03646815d8"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.417106 8301 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:30.418442 8301 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 22:17:30.418509 8301 log.cc:1196] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 22:17:30.418807 8301 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Deleting consensus metadata
I20260923 22:17:30.418937 8301 master_runner.cc:315] Copying system tablet from 127.1.129.126:37359
I20260923 22:17:30.419507 8301 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: tablet copy: Beginning tablet copy session from remote peer at address 127.1.129.126:37359
I20260923 22:17:30.426661 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.419766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37164)
0923 22:17:30.419896 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.419906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.419966 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.420117 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.420118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.420305 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.420355 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.420357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.421306 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.421307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.422114 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.422116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.422595 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.422596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.422668 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.423219 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.423224 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.423637 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.425524 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.425525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.425526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.425741 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.425964 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.425965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.425965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.426030 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.426314 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.426315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.426361 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.426431 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.426530 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260923 22:17:30.426710 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.419860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.1.129.126:37359)
0923 22:17:30.419917 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.419919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.419954 (+ 35us) server_negotiation.cc:407] Connection header received
0923 22:17:30.420020 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.420021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.420031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.420111 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.420449 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.421214 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.422212 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.422511 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.422590 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.423758 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.423760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.423760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.423768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.425436 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.425824 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.425826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.425826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.425890 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.426101 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.426102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.426103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.426236 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.426311 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.426477 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.426558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:17:30.427109 7669 tablet_copy_service.cc:140] P 6ff3279a277842738477b74a18bf8336: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a4d323ca0bf345db9234c194b536f76e ({username='oryx', principal='oryx/127.1.129.124@KRBTEST.COM'} at 127.0.0.1:37164)
I20260923 22:17:30.427169 7669 tablet_copy_service.cc:161] P 6ff3279a277842738477b74a18bf8336: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a4d323ca0bf345db9234c194b536f76e at {username='oryx', principal='oryx/127.1.129.124@KRBTEST.COM'} at 127.0.0.1:37164: session id = a4d323ca0bf345db9234c194b536f76e-00000000000000000000000000000000
I20260923 22:17:30.427768 7669 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 22:17:30.428256 8301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 22:17:30.429513 8301 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: tablet copy: Starting download of 0 data blocks...
I20260923 22:17:30.429663 8301 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: tablet copy: Starting download of 1 WAL segments...
I20260923 22:17:30.430411 8301 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 22:17:30.431392 7669 tablet_copy_service.cc:342] P 6ff3279a277842738477b74a18bf8336: Request end of tablet copy session a4d323ca0bf345db9234c194b536f76e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.129.124@KRBTEST.COM'} at 127.0.0.1:37164
I20260923 22:17:30.431450 7669 tablet_copy_service.cc:433] P 6ff3279a277842738477b74a18bf8336: ending tablet copy session a4d323ca0bf345db9234c194b536f76e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a4d323ca0bf345db9234c194b536f76e
I20260923 22:17:30.434337 8301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.435526 8407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.437222 8301 server_base.cc:1034] running on GCE node
W20260923 22:17:30.437184 8410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.446453 8408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.446612 8301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.446767 8301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.447893 8301 hybrid_clock.cc:648] HybridClock initialized: now 1790201850447881 us; error 32 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.449465 8301 init.cc:377] Logged in from keytab as oryx/127.1.129.124@KRBTEST.COM (short username oryx)
I20260923 22:17:30.449731 8301 webserver.cc:460] Webserver started at http://127.1.129.124:39541/ using document root <none> and password file <none>
I20260923 22:17:30.449853 8301 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.449903 8301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.450748 8301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.451351 8417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.451502 8301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:30.451553 8301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
uuid: "a4d323ca0bf345db9234c194b536f76e"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-dmw7"
server_key: "10c42e61bd3821bae43db222a5c69a2b"
server_key_iv: "a5cd738e55d4a1185702aa03646815d8"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.451607 8301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.462945 8301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.464769 8301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.467912 8301 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.124:33931
I20260923 22:17:30.467993 8469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.124:33931 every 8 connection(s)
I20260923 22:17:30.468238 8301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
I20260923 22:17:30.469184 8470 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:17:30.469750 8470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Bootstrap starting.
I20260923 22:17:30.472698 8470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:17:30.472898 8470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: Bootstrap complete.
I20260923 22:17:30.473083 8470 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } }
I20260923 22:17:30.473166 8470 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4d323ca0bf345db9234c194b536f76e, State: Initialized, Role: LEARNER
I20260923 22:17:30.473201 8470 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } }
I20260923 22:17:30.473358 8470 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:30.473364 8473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.473572 8473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:30.475099 8484 catalog_manager.cc:1259] Loaded cluster ID: 9437cbf915884997a3a22450e6fd2837
I20260923 22:17:30.475185 8484 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: loading cluster ID for follower catalog manager: success
I20260923 22:17:30.476305 8484 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: acquiring CA information for follower catalog manager: success
I20260923 22:17:30.476626 8484 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 22:17:30.484678 8395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.484869 8395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.484892 8395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.486029 8395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.486069 8395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.486095 8395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:30.487613 8395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab
--principal=oryx/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:32841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=46439
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633,127.1.129.124:33931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:30.487848 8395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.488085 8395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.493769 8490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.493851 8489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.493782 8492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.494081 8395 server_base.cc:1034] running on GCE node
I20260923 22:17:30.494274 8395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.494511 8395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.495678 8395 hybrid_clock.cc:648] HybridClock initialized: now 1790201850495664 us; error 31 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.498157 8395 init.cc:377] Logged in from keytab as oryx/127.1.129.65@KRBTEST.COM (short username oryx)
I20260923 22:17:30.498672 8395 webserver.cc:460] Webserver started at http://127.1.129.65:46439/ using document root <none> and password file <none>
I20260923 22:17:30.498901 8395 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.498979 8395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.500365 8395 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.500977 8499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.501173 8395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.501279 8395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "d1f50f21c8f849eeb01f9ed22733ac91"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "c586c127198e45c12fe1e9beb7df1014"
server_key_iv: "910fc1bdf7f1012ae6f33714cca03f09"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.501607 8395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.511273 8395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.513214 8395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.513368 8395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.513576 8395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:30.513880 8395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:30.513934 8395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.513975 8395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:30.514003 8395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.520407 8395 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:32841
I20260923 22:17:30.520444 8612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:32841 every 8 connection(s)
I20260923 22:17:30.520787 8395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.125@KRBTEST.COM
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.124@KRBTEST.COM
I20260923 22:17:30.530710 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 8395
I20260923 22:17:30.530864 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 7845
I20260923 22:17:30.532333 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.521905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:41825 (local address 127.1.129.126:37359)
0923 22:17:30.521949 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.521951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.521958 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:30.522958 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.522960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.522975 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.523007 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.524214 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.524864 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.525977 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.526269 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.526305 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.528938 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.528939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.528940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.528946 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.530697 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.531179 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.531180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.531180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.531230 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.531498 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.531500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.531500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.531667 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.531739 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.532102 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.532189 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:30.532903 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.521780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.65:41825)
0923 22:17:30.521883 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.521891 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.522845 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.523092 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.523094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.523530 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.524112 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.524115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.524954 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.524955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.525874 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.525877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.526382 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.526383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.526440 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.527189 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.527193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.528818 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.530783 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.530789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.530791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.531096 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.531291 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.531292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.531292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.531427 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.531741 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.531746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.531943 (+ 197us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.532424 (+ 481us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.532631 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":29,"thread_start_us":56,"threads_started":1}
I20260923 22:17:30.533617 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.521487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.1.129.65:54861)
0923 22:17:30.521738 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.521779 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.522837 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.523083 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.523086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.523511 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.523732 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.523736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.524583 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.524584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.525839 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.525841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.526298 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.526299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.526436 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.527108 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.527118 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.529261 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.532458 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.532459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.532459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.532680 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.532899 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.532900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.532901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.532987 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.533274 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.533275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.533317 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.533388 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.533491 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":17,"thread_start_us":114,"threads_started":1}
I20260923 22:17:30.533669 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.521650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:54861 (local address 127.1.129.125:45633)
0923 22:17:30.521693 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.521695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.521867 (+ 172us) server_negotiation.cc:407] Connection header received
0923 22:17:30.522933 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.522935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.522948 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.523008 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.523825 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.524474 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.525921 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.526212 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.526281 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.529349 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.529351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.529351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.529359 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.531084 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.532769 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.532770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.532771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.532827 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.533047 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.533048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.533048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.533167 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.533247 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.533438 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.533535 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:30.533893 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.521497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.1.129.65:43969)
0923 22:17:30.521747 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.521779 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.522837 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.523104 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.523104 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.523535 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.524129 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.524130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.525191 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.525192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.525960 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.525962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.526387 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.526388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.526446 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.527133 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.527137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.529913 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.532662 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.532663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.532663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.532882 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.533077 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.533078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.533078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.533137 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.533455 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.533455 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.533490 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.533554 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.533631 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":5,"thread_start_us":99,"threads_started":1}
I20260923 22:17:30.534096 8615 heartbeater.cc:347] Connected to a master server at 127.1.129.125:45633
I20260923 22:17:30.534130 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.521536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:43969 (local address 127.1.129.124:33931)
0923 22:17:30.521669 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.521671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.521948 (+ 277us) server_negotiation.cc:407] Connection header received
0923 22:17:30.522989 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.522991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.523004 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.523063 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.524221 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.525084 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.526036 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.526273 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.526335 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.530012 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.530013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.530013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.530022 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.532573 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.532956 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.532957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.532957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.533013 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.533222 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.533224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.533224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.533367 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.533429 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.533993 (+ 564us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.534069 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260923 22:17:30.534197 8615 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.534379 8615 heartbeater.cc:511] Master 127.1.129.125:45633 requested a full tablet report, sending...
I20260923 22:17:30.534956 8247 ts_manager.cc:194] Registered new tserver with Master: d1f50f21c8f849eeb01f9ed22733ac91 (127.1.129.65:32841)
I20260923 22:17:30.536659 8614 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37359
I20260923 22:17:30.536712 8614 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.536775 8614 heartbeater.cc:511] Master 127.1.129.126:37359 requested a full tablet report, sending...
I20260923 22:17:30.536854 8613 heartbeater.cc:347] Connected to a master server at 127.1.129.124:33931
I20260923 22:17:30.536886 8613 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.536942 8613 heartbeater.cc:511] Master 127.1.129.124:33931 requested a full tablet report, sending...
I20260923 22:17:30.537040 7649 ts_manager.cc:194] Re-registered known tserver with Master: d1f50f21c8f849eeb01f9ed22733ac91 (127.1.129.65:32841)
I20260923 22:17:30.537202 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:39403
--local_ip_for_outbound_sockets=127.1.129.66
--tserver_master_addrs=127.1.129.126:37359
--webserver_port=34993
--webserver_interface=127.1.129.66
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab
--principal=oryx/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633,127.1.129.124:33931 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:17:30.537266 8434 ts_manager.cc:194] Registered new tserver with Master: d1f50f21c8f849eeb01f9ed22733ac91 (127.1.129.65:32841)
I20260923 22:17:30.537555 7649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:41825
W20260923 22:17:30.617337 8621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.617523 8621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.617544 8621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.618813 8621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.618850 8621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.618876 8621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:30.620425 8621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab
--principal=oryx/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:39403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=34993
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633,127.1.129.124:33931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:30.620682 8621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.620921 8621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:30.626011 8621 server_base.cc:1034] running on GCE node
W20260923 22:17:30.625953 8627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.625946 8626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.625990 8629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.626452 8621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.626744 8621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.627911 8621 hybrid_clock.cc:648] HybridClock initialized: now 1790201850627896 us; error 26 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.630477 8621 init.cc:377] Logged in from keytab as oryx/127.1.129.66@KRBTEST.COM (short username oryx)
I20260923 22:17:30.630947 8621 webserver.cc:460] Webserver started at http://127.1.129.66:34993/ using document root <none> and password file <none>
I20260923 22:17:30.631196 8621 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.631279 8621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.632588 8621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:30.633155 8636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.633323 8621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:30.633409 8621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "3d7571de4fee408e94e2d7001258bd55"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "b5362735e02de3f1a5910f805f030e6e"
server_key_iv: "289e104f559d899195f98c72702b45ea"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.633730 8621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.640113 8621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.642295 8621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.642458 8621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.642701 8621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:30.642982 8621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:30.643014 8621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.643061 8621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:30.643088 8621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.648851 8621 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:39403
I20260923 22:17:30.648896 8749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:39403 every 8 connection(s)
I20260923 22:17:30.649217 8621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:30.652130 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 8621
I20260923 22:17:30.652297 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 7979
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.124@KRBTEST.COM
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:30.658608 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:36269
--local_ip_for_outbound_sockets=127.1.129.67
--tserver_master_addrs=127.1.129.126:37359
--webserver_port=36815
--webserver_interface=127.1.129.67
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab
--principal=oryx/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633,127.1.129.124:33931 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.125@KRBTEST.COM
I20260923 22:17:30.661836 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.650335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:39131 (local address 127.1.129.124:33931)
0923 22:17:30.650395 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.650397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.650458 (+ 61us) server_negotiation.cc:407] Connection header received
0923 22:17:30.651286 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.651287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.651299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.651331 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.654145 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.654754 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.655476 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.655692 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.655775 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.658235 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.658236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.658237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.658245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.660150 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.660755 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.660756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.660757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.660816 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.661082 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.661083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.661083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.661231 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.661275 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.661649 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.661738 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:30.661943 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.650218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.1.129.66:39131)
0923 22:17:30.650409 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.650419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.651189 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.651432 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.651433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.651847 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.654053 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.654055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.654849 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.654850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.655373 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.655374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.655776 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.655777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.655850 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.656283 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.656303 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.658127 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.660238 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.660244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.660246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.660662 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.660888 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.660889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.660889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.661011 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.661323 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.661329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.661495 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.661611 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.661759 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":19,"thread_start_us":58,"threads_started":1}
I20260923 22:17:30.662153 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.649916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:37335)
0923 22:17:30.650116 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.650148 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.651169 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.651415 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.651418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.651861 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.652059 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.652063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.653014 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.653014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.653794 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.653796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.654190 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.654190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.654339 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.656333 (+ 1994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.656338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.658131 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.660437 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.660438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.660438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.660965 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.661183 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.661184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.661185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.661246 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.661793 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.661794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.661847 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.661919 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.662022 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":14,"thread_start_us":90,"threads_started":1}
I20260923 22:17:30.662379 8750 heartbeater.cc:347] Connected to a master server at 127.1.129.124:33931
I20260923 22:17:30.662457 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.649983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:37335 (local address 127.1.129.126:37359)
0923 22:17:30.650027 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.650029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.650304 (+ 275us) server_negotiation.cc:407] Connection header received
0923 22:17:30.651283 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.651285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.651298 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.651348 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.652152 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.652894 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.653888 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.654119 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.654187 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.658275 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.658277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.658277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.658285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.660349 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.661047 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.661048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.661048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.661111 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.661320 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.661321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.661321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.661459 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.661762 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.662274 (+ 512us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.662353 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:30.662470 8750 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.662604 8750 heartbeater.cc:511] Master 127.1.129.124:33931 requested a full tablet report, sending...
I20260923 22:17:30.663169 8751 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37359
I20260923 22:17:30.663220 8751 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.663290 8751 heartbeater.cc:511] Master 127.1.129.126:37359 requested a full tablet report, sending...
I20260923 22:17:30.663486 8434 ts_manager.cc:194] Registered new tserver with Master: 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403)
I20260923 22:17:30.663544 7649 ts_manager.cc:194] Re-registered known tserver with Master: 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403)
I20260923 22:17:30.663929 7649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:37335
I20260923 22:17:30.670557 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.649919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:49181 (local address 127.1.129.125:45633)
0923 22:17:30.650039 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.650041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.650285 (+ 244us) server_negotiation.cc:407] Connection header received
0923 22:17:30.651286 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.651287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.651299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.651331 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.652169 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.652946 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.657911 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.658199 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.658279 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.660599 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.660601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.660601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.660608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.662598 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.669770 (+ 7172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.669771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.669771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.669829 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.670096 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.670097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.670098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.670217 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.670270 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.670432 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.670496 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 22:17:30.670608 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.649916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.1.129.66:49181)
0923 22:17:30.650144 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.650155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.651195 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.651422 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.651423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.651831 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.652061 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.652067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.655204 (+ 3137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.655205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.656191 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.656192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.658509 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.658510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.658566 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.659106 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.659111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.660513 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.669420 (+ 8907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.669422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.669423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.669667 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.669913 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.669913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.669914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.670007 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.670284 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.670285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.670328 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.670394 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.670500 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":38,"thread_start_us":86,"threads_started":1}
I20260923 22:17:30.671015 8752 heartbeater.cc:347] Connected to a master server at 127.1.129.125:45633
I20260923 22:17:30.671084 8752 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.671169 8752 heartbeater.cc:511] Master 127.1.129.125:45633 requested a full tablet report, sending...
I20260923 22:17:30.671435 8247 ts_manager.cc:194] Registered new tserver with Master: 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403)
W20260923 22:17:30.741992 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.742208 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.742247 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.743456 8757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.743520 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.743564 8757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:30.745013 8757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab
--principal=oryx/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:36269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=36815
--tserver_master_addrs=127.1.129.126:37359,127.1.129.125:45633,127.1.129.124:33931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:30.745281 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.745509 8757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.750557 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.750563 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.750563 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.750840 8757 server_base.cc:1034] running on GCE node
I20260923 22:17:30.751030 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.751314 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.752513 8757 hybrid_clock.cc:648] HybridClock initialized: now 1790201850752501 us; error 26 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.754892 8757 init.cc:377] Logged in from keytab as oryx/127.1.129.67@KRBTEST.COM (short username oryx)
I20260923 22:17:30.755336 8757 webserver.cc:460] Webserver started at http://127.1.129.67:36815/ using document root <none> and password file <none>
I20260923 22:17:30.755573 8757 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.755654 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.756953 8757 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:30.757565 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.757746 8757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:30.757866 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "c7c59dfe33c347d690d5f0ce962522f6"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-dmw7"
server_key: "0eb66be4dfb77337a02352e1461d5757"
server_key_iv: "b1e20f010ba95d3e6de2ef6db13ca7a5"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.758180 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.778391 8757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.780452 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.780615 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.780853 8757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:30.781157 8757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:30.781210 8757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.781250 8757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:30.781278 8757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.787153 8757 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:36269
I20260923 22:17:30.787192 8885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:36269 every 8 connection(s)
I20260923 22:17:30.787551 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.124@KRBTEST.COM
Sep 23 22:17:30 dist-test-slave-dmw7 krb5kdc[7604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.125@KRBTEST.COM
I20260923 22:17:30.796118 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 8757
I20260923 22:17:30.799028 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.788289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:35373 (local address 127.1.129.124:33931)
0923 22:17:30.788377 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.788380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.788612 (+ 232us) server_negotiation.cc:407] Connection header received
0923 22:17:30.789378 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.789380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.789391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.789446 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.790301 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.791070 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792393 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.792679 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792737 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.795961 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.795963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.795963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.795971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.797451 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.797980 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.797981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.797982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.798040 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.798281 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.798282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.798282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.798417 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.798483 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.798851 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.798936 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:17:30.799142 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.788642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:47691 (local address 127.1.129.126:37359)
0923 22:17:30.788697 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.788700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.788798 (+ 98us) server_negotiation.cc:407] Connection header received
0923 22:17:30.789578 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.789579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.789591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.789687 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.790295 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.790951 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792423 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.792629 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792706 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.795469 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.795470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.795471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.795477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.797961 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.798331 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.798332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.798333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.798384 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.798589 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.798590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.798591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.798752 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.798814 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.798985 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.799056 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:30.799588 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.788274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.1.129.67:35373)
0923 22:17:30.788463 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.788499 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.789205 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.789536 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.789541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.790017 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.790188 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.790194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.791155 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.791156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.792311 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.792313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.792782 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.792838 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.793654 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.793658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.795862 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.797557 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.797565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.797567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.797885 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.798111 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.798112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.798112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.798211 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.798505 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.798510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.798687 (+ 177us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.799117 (+ 430us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.799348 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":123,"thread_start_us":107,"threads_started":1}
I20260923 22:17:30.799588 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.788555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:47691)
0923 22:17:30.788727 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.788736 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.789208 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.789721 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.789722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.789983 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.790188 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.790193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.791058 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.791059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.792317 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.792318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.792705 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.792826 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.793724 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.793729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.795379 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.798044 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.798045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.798046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.798266 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.798458 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.798459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.798459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.798517 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.798839 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.798840 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.798875 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.799217 (+ 342us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.799394 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":177,"thread_start_us":71,"threads_started":1}
I20260923 22:17:30.799947 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.788986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:52021 (local address 127.1.129.125:45633)
0923 22:17:30.789044 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.789046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.789054 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:30.789303 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.789304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.789315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.789441 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.790301 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.791110 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792290 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.792571 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792631 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.796538 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.796540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.796540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.796548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.798791 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.799186 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.799187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.799188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.799251 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.799449 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.799450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.799450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.799563 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.799600 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.799770 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.799856 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:30.800338 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.788810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.1.129.67:52021)
0923 22:17:30.788950 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.788959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.789214 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.789914 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.789915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.790091 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.790188 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.790194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.791181 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.791182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.792192 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.792194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.792696 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.792697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.792838 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.793576 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.793584 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.796438 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.798888 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.798890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.798890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.799100 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.799328 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.799329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.799329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.799385 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.799638 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.799639 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.799675 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.800128 (+ 453us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.800228 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":94,"thread_start_us":79,"threads_started":1}
I20260923 22:17:30.800673 8888 heartbeater.cc:347] Connected to a master server at 127.1.129.125:45633
I20260923 22:17:30.800750 8887 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37359
I20260923 22:17:30.800750 8888 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.800805 8887 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.800989 8887 heartbeater.cc:511] Master 127.1.129.126:37359 requested a full tablet report, sending...
I20260923 22:17:30.800989 8888 heartbeater.cc:511] Master 127.1.129.125:45633 requested a full tablet report, sending...
I20260923 22:17:30.801414 7649 ts_manager.cc:194] Re-registered known tserver with Master: c7c59dfe33c347d690d5f0ce962522f6 (127.1.129.67:36269)
I20260923 22:17:30.801476 8247 ts_manager.cc:194] Registered new tserver with Master: c7c59dfe33c347d690d5f0ce962522f6 (127.1.129.67:36269)
I20260923 22:17:30.801873 7649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:47691
I20260923 22:17:30.801954 8886 heartbeater.cc:347] Connected to a master server at 127.1.129.124:33931
I20260923 22:17:30.801990 8886 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.802060 8886 heartbeater.cc:511] Master 127.1.129.124:33931 requested a full tablet report, sending...
I20260923 22:17:30.803078 8434 ts_manager.cc:194] Registered new tserver with Master: c7c59dfe33c347d690d5f0ce962522f6 (127.1.129.67:36269)
I20260923 22:17:30.806977 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.797698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.1.129.125:45633)
0923 22:17:30.800021 (+ 2323us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.800022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.800029 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:30.800045 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.800046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.800055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.800105 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.800811 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.801368 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.802160 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.802407 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.802438 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.804077 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.804078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.804078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.804087 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.805718 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.806132 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.806134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.806134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.806185 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.806408 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.806408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.806409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.806519 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.806613 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.806796 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.806878 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2310,"thread_start_us":50,"threads_started":1}
I20260923 22:17:30.810427 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.798731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.1.129.126:37359)
0923 22:17:30.799201 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.799203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.803037 (+ 3834us) server_negotiation.cc:407] Connection header received
0923 22:17:30.803143 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.803144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.803188 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.803220 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.803958 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.804648 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.805571 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.805811 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.805871 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.807720 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.807721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.807721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.807726 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.809236 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.809667 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.809668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.809669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.809724 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.809924 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.809925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.809926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.810050 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.810133 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.810261 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.810337 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":76,"threads_started":1}
I20260923 22:17:30.810598 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.799095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.1.129.124:33931)
0923 22:17:30.802378 (+ 3283us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.802380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.803080 (+ 700us) server_negotiation.cc:407] Connection header received
0923 22:17:30.803237 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.803237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.803249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.803277 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.803849 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.804577 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.805762 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.806019 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.806097 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.807971 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.807972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.807972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.807979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.809341 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.809725 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.809725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.809726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.809794 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.810034 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.810035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.810035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.810186 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.810273 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.810413 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.810483 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3263}
I20260923 22:17:30.811940 7649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:30.813380 7649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:30.820609 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.816934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:39403 (local address 127.0.0.1:52642)
0923 22:17:30.817045 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.817056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.817102 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.817534 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.817535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.817537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.817582 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.817584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.818406 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.818407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.819493 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.819495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.820373 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.820374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.820432 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.820437 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.820499 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.820546 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260923 22:17:30.820631 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.817128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.1.129.66:39403)
0923 22:17:30.817375 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.817380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.817389 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.817427 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.817429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.817444 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.817525 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.817644 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.818301 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.819575 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.820294 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.820364 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.820432 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.820525 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.820558 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":77,"threads_started":1}
I20260923 22:17:30.821087 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.816859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.1.129.67:36269)
0923 22:17:30.817319 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.817323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.817332 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.817364 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.817366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.817382 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.817481 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.817595 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.818743 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.820103 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.820780 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.820870 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.820916 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.820989 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.821019 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":66,"threads_started":1}
I20260923 22:17:30.821084 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.816689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:36269 (local address 127.0.0.1:53824)
0923 22:17:30.816728 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.816737 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.816771 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.817481 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.817481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.817483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.817521 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.817522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.818836 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.818838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.819993 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.819996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.820868 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.820869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.820924 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.820927 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.820979 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.821016 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 22:17:30.821255 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.816739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:32841 (local address 127.0.0.1:58212)
0923 22:17:30.816859 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.816869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.816912 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.817196 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.817197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.817199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.817248 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.817250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.818972 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.818973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.819991 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.819993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.820882 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.820883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.820943 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.820947 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.820974 (+ 27us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.821007 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":8,"thread_start_us":83,"threads_started":1}
I20260923 22:17:30.821688 8820 tablet_service.cc:1467] Processing CreateTablet for tablet cb6a92e96f244429a6a9a08e03eafecc (DEFAULT_TABLE table=test-table [id=891789ad52994b7cafbaae53b4540ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:30.821883 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.816820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.1.129.65:32841)
0923 22:17:30.817024 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.817029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.817040 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:30.817083 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.817085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.817102 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.817200 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.817546 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.818900 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.820064 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.820805 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.821058 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.821738 (+ 680us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.821767 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.821807 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260923 22:17:30.821830 8684 tablet_service.cc:1467] Processing CreateTablet for tablet cb6a92e96f244429a6a9a08e03eafecc (DEFAULT_TABLE table=test-table [id=891789ad52994b7cafbaae53b4540ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:30.822011 8820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6a92e96f244429a6a9a08e03eafecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.822104 8684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6a92e96f244429a6a9a08e03eafecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.823194 8547 tablet_service.cc:1467] Processing CreateTablet for tablet cb6a92e96f244429a6a9a08e03eafecc (DEFAULT_TABLE table=test-table [id=891789ad52994b7cafbaae53b4540ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:30.823489 8547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6a92e96f244429a6a9a08e03eafecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.824769 8911 tablet_bootstrap.cc:492] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Bootstrap starting.
I20260923 22:17:30.825282 8912 tablet_bootstrap.cc:492] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: Bootstrap starting.
I20260923 22:17:30.825462 8911 tablet_bootstrap.cc:654] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.825784 8911 log.cc:824] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.825857 8912 tablet_bootstrap.cc:654] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.826115 8912 log.cc:824] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.826553 8911 tablet_bootstrap.cc:492] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: No bootstrap required, opened a new log
I20260923 22:17:30.826669 8911 ts_tablet_manager.cc:1397] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:30.826870 8912 tablet_bootstrap.cc:492] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: No bootstrap required, opened a new log
I20260923 22:17:30.826946 8912 ts_tablet_manager.cc:1397] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:30.827302 8915 tablet_bootstrap.cc:492] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: Bootstrap starting.
I20260923 22:17:30.827908 8915 tablet_bootstrap.cc:654] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.828190 8915 log.cc:824] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.828974 8915 tablet_bootstrap.cc:492] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: No bootstrap required, opened a new log
I20260923 22:17:30.829038 8915 ts_tablet_manager.cc:1397] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:30.829207 8912 raft_consensus.cc:348] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.829349 8912 raft_consensus.cc:374] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.829391 8912 raft_consensus.cc:729] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c59dfe33c347d690d5f0ce962522f6, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.829347 8911 raft_consensus.cc:348] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.829501 8911 raft_consensus.cc:374] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.829537 8911 raft_consensus.cc:729] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7571de4fee408e94e2d7001258bd55, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.829497 8912 consensus_queue.cc:260] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.829669 8911 consensus_queue.cc:260] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.829754 8912 ts_tablet_manager.cc:1428] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:17:30.829911 8911 ts_tablet_manager.cc:1428] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:30.829988 8887 heartbeater.cc:503] Master 127.1.129.126:37359 was elected leader, sending a full tablet report...
I20260923 22:17:30.830027 8751 heartbeater.cc:503] Master 127.1.129.126:37359 was elected leader, sending a full tablet report...
I20260923 22:17:30.831940 8915 raft_consensus.cc:348] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.832052 8915 raft_consensus.cc:374] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.832072 8915 raft_consensus.cc:729] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f50f21c8f849eeb01f9ed22733ac91, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.832172 8915 consensus_queue.cc:260] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.832379 8915 ts_tablet_manager.cc:1428] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:30.832445 8614 heartbeater.cc:503] Master 127.1.129.126:37359 was elected leader, sending a full tablet report...
W20260923 22:17:30.899927 8753 tablet.cc:2369] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:30.938482 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.935936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.1.129.124:33931)
0923 22:17:30.936013 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.936015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.936024 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.936048 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.936049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.936079 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.936153 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.936287 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.936769 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.937778 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.938165 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.938268 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.938365 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.938383 (+ 18us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.938417 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 22:17:30.938484 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.935854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.0.0.1:32840)
0923 22:17:30.935916 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.935927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.935985 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.936156 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.936157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.936158 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.936217 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.936219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.936856 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.936857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.937687 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.937689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.938248 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.938249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.938308 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.938312 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.938354 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.938405 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:17:30.938951 8444 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260923 22:17:30.939826 8473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff3279a277842738477b74a18bf8336. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.939930 8473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:30.946188 8917 raft_consensus.cc:482] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:30.946280 8917 raft_consensus.cc:504] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.946698 8917 leader_election.cc:288] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f50f21c8f849eeb01f9ed22733ac91 (127.1.129.65:32841), 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403)
I20260923 22:17:30.949677 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.946879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:52361 (local address 127.1.129.65:32841)
0923 22:17:30.946923 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.946925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.947065 (+ 140us) server_negotiation.cc:407] Connection header received
0923 22:17:30.947093 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.947095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.947105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.947206 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.947362 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.947879 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.948968 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.949325 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.949417 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.949426 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.949574 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.949609 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:30.949679 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.946961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:43461 (local address 127.1.129.66:39403)
0923 22:17:30.947002 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.947003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.947013 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:30.947043 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.947043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.947052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.947111 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.947263 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.947886 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.948969 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.949317 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.949417 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.949426 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.949570 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.949608 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:17:30.949719 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.946838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:32841 (local address 127.1.129.67:52361)
0923 22:17:30.946894 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.946906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.946986 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.947209 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.947210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.947211 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.947277 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.947278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.947957 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.947958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.948873 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.948876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.949417 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.949418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.949492 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.949495 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.949566 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.949608 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":70}
I20260923 22:17:30.949733 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.946857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:39403 (local address 127.1.129.67:43461)
0923 22:17:30.946944 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.946954 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.946992 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.947117 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.947118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.947119 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.947207 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.947209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.947964 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.947965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.948885 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.948887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.949418 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.949419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.949498 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.949501 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.949565 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.949608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":38}
I20260923 22:17:30.950057 8704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6a92e96f244429a6a9a08e03eafecc" candidate_uuid: "c7c59dfe33c347d690d5f0ce962522f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7571de4fee408e94e2d7001258bd55" is_pre_election: true
I20260923 22:17:30.950054 8567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6a92e96f244429a6a9a08e03eafecc" candidate_uuid: "c7c59dfe33c347d690d5f0ce962522f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" is_pre_election: true
I20260923 22:17:30.950201 8704 raft_consensus.cc:2454] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c59dfe33c347d690d5f0ce962522f6 in term 0.
I20260923 22:17:30.950201 8567 raft_consensus.cc:2454] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c59dfe33c347d690d5f0ce962522f6 in term 0.
I20260923 22:17:30.950393 8775 leader_election.cc:302] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7571de4fee408e94e2d7001258bd55, c7c59dfe33c347d690d5f0ce962522f6; no voters:
I20260923 22:17:30.950501 8917 raft_consensus.cc:2781] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:30.950551 8917 raft_consensus.cc:482] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:30.950573 8917 raft_consensus.cc:3037] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.951284 8917 raft_consensus.cc:504] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.951421 8917 leader_election.cc:288] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [CANDIDATE]: Term 1 election: Requested vote from peers d1f50f21c8f849eeb01f9ed22733ac91 (127.1.129.65:32841), 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403)
I20260923 22:17:30.951567 8567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6a92e96f244429a6a9a08e03eafecc" candidate_uuid: "c7c59dfe33c347d690d5f0ce962522f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f50f21c8f849eeb01f9ed22733ac91"
I20260923 22:17:30.951603 8704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6a92e96f244429a6a9a08e03eafecc" candidate_uuid: "c7c59dfe33c347d690d5f0ce962522f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7571de4fee408e94e2d7001258bd55"
I20260923 22:17:30.951640 8567 raft_consensus.cc:3037] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.951695 8704 raft_consensus.cc:3037] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.952338 8567 raft_consensus.cc:2454] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c59dfe33c347d690d5f0ce962522f6 in term 1.
I20260923 22:17:30.952504 8776 leader_election.cc:302] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7c59dfe33c347d690d5f0ce962522f6, d1f50f21c8f849eeb01f9ed22733ac91; no voters:
I20260923 22:17:30.952533 8704 raft_consensus.cc:2454] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c59dfe33c347d690d5f0ce962522f6 in term 1.
I20260923 22:17:30.952605 8917 raft_consensus.cc:2781] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:30.952772 8917 raft_consensus.cc:686] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 1 LEADER]: Becoming Leader. State: Replica: c7c59dfe33c347d690d5f0ce962522f6, State: Running, Role: LEADER
I20260923 22:17:30.952885 8917 consensus_queue.cc:237] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:30.953814 7648 catalog_manager.cc:5495] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7c59dfe33c347d690d5f0ce962522f6 (127.1.129.67). New cstate: current_term: 1 leader_uuid: "c7c59dfe33c347d690d5f0ce962522f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:30.954353 8398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: NON_VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: true } } }
I20260923 22:17:30.954455 8398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:30.954977 8398 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: attempting to promote NON_VOTER a4d323ca0bf345db9234c194b536f76e to VOTER
I20260923 22:17:30.955123 8398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } }
I20260923 22:17:30.955560 8256 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Refusing update from remote peer 6ff3279a277842738477b74a18bf8336: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260923 22:17:30.955708 8443 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 LEARNER]: Refusing update from remote peer 6ff3279a277842738477b74a18bf8336: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260923 22:17:30.955940 8925 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260923 22:17:30.956058 8926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260923 22:17:30.956542 8925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a4d323ca0bf345db9234c194b536f76e (127.1.129.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:30.956692 8256 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a4d323ca0bf345db9234c194b536f76e (127.1.129.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:30.957317 8398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:30.957365 8925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:30.957401 8398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:30.957460 8398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:30.957499 8398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:30.957410 8925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:30.957502 8443 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a4d323ca0bf345db9234c194b536f76e (127.1.129.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:30.957680 8928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:30.957777 8928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:30.958354 8473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ff3279a277842738477b74a18bf8336" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:30.958444 8473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:30.966679 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.964166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.1.129.67:36269)
0923 22:17:30.964213 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.964215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.964224 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.964337 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.964338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.964358 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.964437 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:30.964579 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.965290 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.965963 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.966140 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.966192 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.966321 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:30.966409 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:30.966464 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.966552 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.966566 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:17:30.968333 8704 raft_consensus.cc:1261] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 1 FOLLOWER]: Refusing update from remote peer c7c59dfe33c347d690d5f0ce962522f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:30.968329 8567 raft_consensus.cc:1261] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 1 FOLLOWER]: Refusing update from remote peer c7c59dfe33c347d690d5f0ce962522f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:30.968683 8917 consensus_queue.cc:1035] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:30.968811 8923 consensus_queue.cc:1035] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:30.971091 8930 mvcc.cc:205] Tried to move back new op lower bound from 7332666781563691008 to 7332666781503590400. Current Snapshot: MvccSnapshot[applied={T|T < 7332666781563691008}]
I20260923 22:17:30.971299 8931 mvcc.cc:205] Tried to move back new op lower bound from 7332666781563691008 to 7332666781503590400. Current Snapshot: MvccSnapshot[applied={T|T < 7332666781563691008}]
I20260923 22:17:30.975772 7649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:30.975876 7649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:30.977115 7649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff3279a277842738477b74a18bf8336: Sending DeleteTablet for 3 replicas of tablet cb6a92e96f244429a6a9a08e03eafecc
I20260923 22:17:30.977468 8684 tablet_service.cc:1514] Processing DeleteTablet for tablet cb6a92e96f244429a6a9a08e03eafecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:30 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:52642
I20260923 22:17:30.977471 8547 tablet_service.cc:1514] Processing DeleteTablet for tablet cb6a92e96f244429a6a9a08e03eafecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:30 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:58212
I20260923 22:17:30.977849 8820 tablet_service.cc:1514] Processing DeleteTablet for tablet cb6a92e96f244429a6a9a08e03eafecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:30 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:53824
I20260923 22:17:30.978024 8941 tablet_replica.cc:330] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: stopping tablet replica
I20260923 22:17:30.978129 8941 raft_consensus.cc:2229] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:30.978251 8941 raft_consensus.cc:2258] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:30.978754 8940 tablet_replica.cc:330] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: stopping tablet replica
I20260923 22:17:30.978849 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 7617
I20260923 22:17:30.978863 8940 raft_consensus.cc:2229] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:30.979692 8941 ts_tablet_manager.cc:1905] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:30.981325 8941 ts_tablet_manager.cc:1918] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:30.981405 8941 log.cc:1196] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/cb6a92e96f244429a6a9a08e03eafecc
I20260923 22:17:30.981683 8941 ts_tablet_manager.cc:1939] T cb6a92e96f244429a6a9a08e03eafecc P d1f50f21c8f849eeb01f9ed22733ac91: Deleting consensus metadata
I20260923 22:17:30.982125 8940 raft_consensus.cc:2258] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:30.982324 8942 tablet_replica.cc:330] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: stopping tablet replica
I20260923 22:17:30.982409 8942 raft_consensus.cc:2229] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:30.982514 8942 raft_consensus.cc:2258] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:30.982934 8940 ts_tablet_manager.cc:1905] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:30.984480 8940 ts_tablet_manager.cc:1918] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:30.984550 8940 log.cc:1196] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/cb6a92e96f244429a6a9a08e03eafecc
I20260923 22:17:30.984812 8940 ts_tablet_manager.cc:1939] T cb6a92e96f244429a6a9a08e03eafecc P 3d7571de4fee408e94e2d7001258bd55: Deleting consensus metadata
I20260923 22:17:30.985458 8942 ts_tablet_manager.cc:1905] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 22:17:30.985723 8503 connection.cc:535] server connection from 127.0.0.1:58212 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 22:17:30.985746 8637 connection.cc:535] server connection from 127.0.0.1:52642 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 22:17:30.987182 8942 ts_tablet_manager.cc:1918] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:30.987263 8942 log.cc:1196] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/cb6a92e96f244429a6a9a08e03eafecc
I20260923 22:17:30.987509 8942 ts_tablet_manager.cc:1939] T cb6a92e96f244429a6a9a08e03eafecc P c7c59dfe33c347d690d5f0ce962522f6: Deleting consensus metadata
W20260923 22:17:30.987849 8774 connection.cc:441] server connection from 127.0.0.1:53824 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53824 (request call id 1) could send its response
I20260923 22:17:30.988284 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.978407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32844 (local address 127.1.129.124:33931)
0923 22:17:30.978439 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.978442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.978449 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:30.978496 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.978497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.978507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.978536 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.980023 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.980798 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.982103 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.982448 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.982486 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.984435 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.984436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.984437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.984444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.986652 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.987195 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.987196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.987197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.987255 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.987621 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.987622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.987622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.987760 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.987833 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.988113 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.988193 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:17:30.995507 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.988185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.1.129.125:45633)
0923 22:17:30.988385 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.988387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.988516 (+ 129us) server_negotiation.cc:407] Connection header received
0923 22:17:30.988603 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.988604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.988612 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.988639 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.989403 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.989864 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.990607 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.990847 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.990922 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.992212 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.992213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.992213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.992220 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.993967 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.994364 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.994366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.994366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.994425 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.994941 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.994942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.994942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.995077 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.995193 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.995322 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.995401 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260923 22:17:30.995666 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.987736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.1.129.124:33931)
0923 22:17:30.988278 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.988279 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.988287 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:30.988306 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.988307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.988318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.988389 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.988984 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.989746 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.990867 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.991173 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.991241 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.992587 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.992588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.992589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.992596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.994426 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.994778 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.994779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.994779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.994826 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.995046 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.995047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.995048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.995183 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.995282 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.995392 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.995473 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":48,"threads_started":1}
I20260923 22:17:31.973412 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.973129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.65:36983)
0923 22:17:31.973272 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.973358 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260923 22:17:31.973659 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.973410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:42193)
0923 22:17:31.973526 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.973595 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20260923 22:17:31.973831 8751 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:37359 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
W20260923 22:17:31.974095 8614 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:37359 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
I20260923 22:17:31.974272 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.973894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:60013)
0923 22:17:31.974132 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.974229 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":47,"threads_started":1}
W20260923 22:17:31.974452 8887 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:37359 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
I20260923 22:17:32.477568 8954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ff3279a277842738477b74a18bf8336)
I20260923 22:17:32.477754 8954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } }
I20260923 22:17:32.478089 8955 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ff3279a277842738477b74a18bf8336)
I20260923 22:17:32.478189 8954 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359), a4d323ca0bf345db9234c194b536f76e (127.1.129.124:33931)
I20260923 22:17:32.478217 8955 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } }
I20260923 22:17:32.478479 8955 leader_election.cc:288] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359), 3a0956b8965348e587ec5485f0ce8064 (127.1.129.125:45633)
I20260923 22:17:32.478829 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.478590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37684)
0923 22:17:32.478735 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.478784 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260923 22:17:32.478873 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.478263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37678)
0923 22:17:32.478783 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.478829 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":47,"threads_started":1}
W20260923 22:17:32.478993 8233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
W20260923 22:17:32.479066 8420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
I20260923 22:17:32.479408 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.479301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37698)
0923 22:17:32.479359 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.479383 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260923 22:17:32.479486 8233 leader_election.cc:334] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359): Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
I20260923 22:17:32.479620 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.479437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37712)
0923 22:17:32.479573 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.479597 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260923 22:17:32.479704 8420 leader_election.cc:334] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359): Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
I20260923 22:17:32.481738 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.478719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.1.129.125:45633)
0923 22:17:32.478940 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:32.478942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:32.478949 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:32.478973 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:32.478974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:32.478984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:32.479046 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:32.479262 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.480009 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.480995 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.481370 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.481462 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:32.481474 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:32.481613 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 22:17:32.481652 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":38,"threads_started":1}
I20260923 22:17:32.481739 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.478641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45633 (local address 127.0.0.1:40042)
0923 22:17:32.478792 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.478802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:32.478857 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:32.479093 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:32.479094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:32.479097 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:32.479149 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:32.479151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.480116 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.480116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:32.480885 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:32.480887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.481464 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.481465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:32.481530 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:32.481533 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:32.481606 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 22:17:32.481652 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":54,"threads_started":1}
I20260923 22:17:32.481876 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.478408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:33931 (local address 127.0.0.1:32872)
0923 22:17:32.478840 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.478850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:32.478898 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:32.479128 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:32.479129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:32.479131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:32.479187 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:32.479189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.480044 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.480045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:32.480887 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:32.480890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.481495 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.481497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:32.481557 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:32.481569 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 22:17:32.481643 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:17:32.481688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":46,"threads_started":1}
I20260923 22:17:32.481812 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.478714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.1.129.124:33931)
0923 22:17:32.478995 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:32.478997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:32.479004 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:32.479024 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:32.479025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:32.479036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:32.479252 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:32.479285 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.479974 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.480989 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.481400 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.481483 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:32.481494 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:32.481650 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 22:17:32.481696 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":35,"threads_started":1}
I20260923 22:17:32.482275 8443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a0956b8965348e587ec5485f0ce8064" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a4d323ca0bf345db9234c194b536f76e" is_pre_election: true
I20260923 22:17:32.482375 8443 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0956b8965348e587ec5485f0ce8064 in term 1.
I20260923 22:17:32.482353 8256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4d323ca0bf345db9234c194b536f76e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a0956b8965348e587ec5485f0ce8064" is_pre_election: true
I20260923 22:17:32.482431 8256 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d323ca0bf345db9234c194b536f76e in term 1.
I20260923 22:17:32.482532 8233 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a0956b8965348e587ec5485f0ce8064, a4d323ca0bf345db9234c194b536f76e; no voters: 6ff3279a277842738477b74a18bf8336
I20260923 22:17:32.482604 8419 leader_election.cc:302] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a0956b8965348e587ec5485f0ce8064, a4d323ca0bf345db9234c194b536f76e; no voters: 6ff3279a277842738477b74a18bf8336
I20260923 22:17:32.482620 8954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 22:17:32.482681 8954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ff3279a277842738477b74a18bf8336)
I20260923 22:17:32.482702 8954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:17:32.482796 8955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 22:17:32.482868 8955 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ff3279a277842738477b74a18bf8336)
I20260923 22:17:32.482898 8955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:17:32.483467 8954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } }
I20260923 22:17:32.483592 8954 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 2 election: Requested vote from peers 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359), a4d323ca0bf345db9234c194b536f76e (127.1.129.124:33931)
I20260923 22:17:32.483561 8955 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } }
I20260923 22:17:32.483688 8955 leader_election.cc:288] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [CANDIDATE]: Term 2 election: Requested vote from peers 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359), 3a0956b8965348e587ec5485f0ce8064 (127.1.129.125:45633)
I20260923 22:17:32.483827 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.483741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37724)
0923 22:17:32.483773 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.483799 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260923 22:17:32.483875 8256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4d323ca0bf345db9234c194b536f76e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a0956b8965348e587ec5485f0ce8064"
I20260923 22:17:32.483916 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.483668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37716)
0923 22:17:32.483867 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.483897 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260923 22:17:32.483961 8256 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a4d323ca0bf345db9234c194b536f76e in current term 2: Already voted for candidate 3a0956b8965348e587ec5485f0ce8064 in this term.
I20260923 22:17:32.484047 8443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a0956b8965348e587ec5485f0ce8064" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a4d323ca0bf345db9234c194b536f76e"
I20260923 22:17:32.484125 8443 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a0956b8965348e587ec5485f0ce8064 in current term 2: Already voted for candidate a4d323ca0bf345db9234c194b536f76e in this term.
I20260923 22:17:32.484227 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.484029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37738)
0923 22:17:32.484177 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.484205 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260923 22:17:32.484232 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.484107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37744)
0923 22:17:32.484185 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.484210 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 22:17:32.484308 8420 leader_election.cc:334] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359): Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
W20260923 22:17:32.484311 8233 leader_election.cc:334] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359): Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
I20260923 22:17:32.484346 8420 leader_election.cc:302] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4d323ca0bf345db9234c194b536f76e; no voters: 3a0956b8965348e587ec5485f0ce8064, 6ff3279a277842738477b74a18bf8336
I20260923 22:17:32.484369 8233 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a0956b8965348e587ec5485f0ce8064; no voters: 6ff3279a277842738477b74a18bf8336, a4d323ca0bf345db9234c194b536f76e
I20260923 22:17:32.484447 8954 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 22:17:32.484472 8955 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 22:17:32.974460 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.974208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:49899)
0923 22:17:32.974359 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.974402 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260923 22:17:32.974833 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.974366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.65:45387)
0923 22:17:32.974532 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.974786 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":48,"threads_started":1}
I20260923 22:17:32.975020 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.974814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:60569)
0923 22:17:32.974945 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.974981 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":52,"threads_started":1}
I20260923 22:17:33.975399 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:33.975163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.65:47291)
0923 22:17:33.975281 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:33.975331 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260923 22:17:33.975427 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:33.974980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:43009)
0923 22:17:33.975337 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:33.975380 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":74,"threads_started":1}
I20260923 22:17:33.975633 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:33.975425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:58909)
0923 22:17:33.975560 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:33.975589 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":41,"threads_started":1}
I20260923 22:17:34.086383 8970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:34.086469 8970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } }
I20260923 22:17:34.086686 8970 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359), a4d323ca0bf345db9234c194b536f76e (127.1.129.124:33931)
I20260923 22:17:34.087041 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.086824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37940)
0923 22:17:34.086949 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.086989 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260923 22:17:34.087096 8443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a0956b8965348e587ec5485f0ce8064" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a4d323ca0bf345db9234c194b536f76e" is_pre_election: true
I20260923 22:17:34.087214 8443 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0956b8965348e587ec5485f0ce8064 in term 2.
I20260923 22:17:34.087448 8233 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0956b8965348e587ec5485f0ce8064, a4d323ca0bf345db9234c194b536f76e; no voters:
I20260923 22:17:34.087481 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.087394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37948)
0923 22:17:34.087438 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.087464 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:17:34.087543 8233 leader_election.cc:334] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359): Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
I20260923 22:17:34.087579 8970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 22:17:34.087608 8970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:34.087636 8970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 2 FOLLOWER]: Advancing to term 3
I20260923 22:17:34.088281 8970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } }
I20260923 22:17:34.088407 8970 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 3 election: Requested vote from peers 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359), a4d323ca0bf345db9234c194b536f76e (127.1.129.124:33931)
I20260923 22:17:34.088569 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.088485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37962)
0923 22:17:34.088525 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.088551 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260923 22:17:34.088656 8443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a0956b8965348e587ec5485f0ce8064" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a4d323ca0bf345db9234c194b536f76e"
I20260923 22:17:34.088732 8443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 2 FOLLOWER]: Advancing to term 3
I20260923 22:17:34.088961 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.088839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37970)
0923 22:17:34.088913 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.088941 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 22:17:34.089046 8233 leader_election.cc:334] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359): Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
I20260923 22:17:34.089491 8443 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a0956b8965348e587ec5485f0ce8064 in term 3.
I20260923 22:17:34.089655 8233 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a0956b8965348e587ec5485f0ce8064, a4d323ca0bf345db9234c194b536f76e; no voters: 6ff3279a277842738477b74a18bf8336
I20260923 22:17:34.089778 8970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 22:17:34.089917 8970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [term 3 LEADER]: Becoming Leader. State: Replica: 3a0956b8965348e587ec5485f0ce8064, State: Running, Role: LEADER
I20260923 22:17:34.090010 8970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } }
I20260923 22:17:34.090368 8973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a0956b8965348e587ec5485f0ce8064. Latest consensus state: current_term: 3 leader_uuid: "3a0956b8965348e587ec5485f0ce8064" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:34.090449 8973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:34.090608 8975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:34.091065 8975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:34.091182 8975 catalog_manager.cc:1259] Loaded cluster ID: 9437cbf915884997a3a22450e6fd2837
I20260923 22:17:34.091217 8975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:34.091399 8975 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:34.091490 8975 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Loaded TSK: 0
I20260923 22:17:34.091631 8975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:34.158245 8247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:34.159969 8247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:34.161082 8443 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [term 3 FOLLOWER]: Refusing update from remote peer 3a0956b8965348e587ec5485f0ce8064: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 22:17:34.161125 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.160968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37980)
0923 22:17:34.161066 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.161094 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 22:17:34.161404 8973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 22:17:34.161614 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.161487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.0.0.1:37986)
0923 22:17:34.161545 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.161584 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260923 22:17:34.161749 8233 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 -> Peer 6ff3279a277842738477b74a18bf8336 (127.1.129.126:37359): Couldn't send request to peer 6ff3279a277842738477b74a18bf8336. Status: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 22:17:34.162063 8977 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a0956b8965348e587ec5485f0ce8064. Latest consensus state: current_term: 3 leader_uuid: "3a0956b8965348e587ec5485f0ce8064" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:34.162186 8977 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:34.162580 8976 mvcc.cc:205] Tried to move back new op lower bound from 7332666794641682432 to 7332666794353295360. Current Snapshot: MvccSnapshot[applied={T|T < 7332666794641682432}]
I20260923 22:17:34.162683 8970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3a0956b8965348e587ec5485f0ce8064" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:34.162765 8970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:34.162910 8978 mvcc.cc:205] Tried to move back new op lower bound from 7332666794641682432 to 7332666794353295360. Current Snapshot: MvccSnapshot[applied={T|T < 7332666794641682432}]
I20260923 22:17:34.163409 8977 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3a0956b8965348e587ec5485f0ce8064" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6ff3279a277842738477b74a18bf8336" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37359 } } peers { permanent_uuid: "3a0956b8965348e587ec5485f0ce8064" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45633 } attrs { promote: false } } peers { permanent_uuid: "a4d323ca0bf345db9234c194b536f76e" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 33931 } attrs { promote: false } } }
I20260923 22:17:34.163678 8977 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d323ca0bf345db9234c194b536f76e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:34.167817 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.164481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.1.129.65:32841)
0923 22:17:34.164589 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.164592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.164638 (+ 46us) server_negotiation.cc:407] Connection header received
0923 22:17:34.164708 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.164708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.164719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.164814 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:34.164979 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.165655 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.166884 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.167472 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.167544 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.167557 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.167690 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.167735 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260923 22:17:34.167832 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.164407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:32841 (local address 127.0.0.1:58222)
0923 22:17:34.164579 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.164589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:34.164650 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:34.164817 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:34.164818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:34.164820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:34.164880 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.164881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.165735 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.165736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.166792 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.166794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.167557 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.167558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.167626 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.167630 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:34.167690 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 22:17:34.167730 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260923 22:17:34.167897 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.164485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:36269 (local address 127.0.0.1:53840)
0923 22:17:34.164828 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.164838 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:34.164881 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:34.165049 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:34.165050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:34.165052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:34.165119 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.165121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.166020 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.166021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.166884 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.166886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.167623 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.167625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.167684 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.167687 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:34.167736 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:17:34.167776 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":54,"thread_start_us":79,"threads_started":1}
I20260923 22:17:34.168179 8547 tablet_service.cc:1467] Processing CreateTablet for tablet 315b26925391427b8abae6cba5fe351b (DEFAULT_TABLE table=test-table [id=b1218dbf8bc846bd866c83987c7fffb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:34.168282 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.164744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.1.129.67:36269)
0923 22:17:34.164865 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.164868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.164878 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:34.164938 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.164938 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.164949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.165093 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:34.165180 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.165925 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.166993 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.167503 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.167620 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.167633 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.168171 (+ 538us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.168213 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260923 22:17:34.168357 8547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315b26925391427b8abae6cba5fe351b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:34.168562 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.164475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:39403 (local address 127.0.0.1:52644)
0923 22:17:34.164611 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.164619 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:34.164666 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:34.165049 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:34.165050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:34.165052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:34.165191 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.165192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.166065 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.166066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.167241 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.167243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.168385 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.168386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.168449 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.168453 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:34.168482 (+ 29us) client_negotiation.cc:240] Negotiation successful
0923 22:17:34.168520 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260923 22:17:34.168546 8820 tablet_service.cc:1467] Processing CreateTablet for tablet 315b26925391427b8abae6cba5fe351b (DEFAULT_TABLE table=test-table [id=b1218dbf8bc846bd866c83987c7fffb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:34.168673 8820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315b26925391427b8abae6cba5fe351b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:34.168883 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.164635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.1.129.66:39403)
0923 22:17:34.164906 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.164908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.164916 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:34.164939 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.164940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.164949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.165249 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:34.165288 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.165984 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.167319 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.168277 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.168739 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.168753 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.168772 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.168815 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":63,"threads_started":1}
I20260923 22:17:34.169073 8684 tablet_service.cc:1467] Processing CreateTablet for tablet 315b26925391427b8abae6cba5fe351b (DEFAULT_TABLE table=test-table [id=b1218dbf8bc846bd866c83987c7fffb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:34.169203 8684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315b26925391427b8abae6cba5fe351b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:34.169999 8987 tablet_bootstrap.cc:492] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91: Bootstrap starting.
I20260923 22:17:34.170145 8988 tablet_bootstrap.cc:492] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: Bootstrap starting.
I20260923 22:17:34.170490 8988 tablet_bootstrap.cc:654] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:34.170485 8987 tablet_bootstrap.cc:654] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:34.170707 8989 tablet_bootstrap.cc:492] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: Bootstrap starting.
I20260923 22:17:34.171096 8987 tablet_bootstrap.cc:492] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91: No bootstrap required, opened a new log
I20260923 22:17:34.171103 8989 tablet_bootstrap.cc:654] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:34.171155 8987 ts_tablet_manager.cc:1397] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:34.171271 8987 raft_consensus.cc:348] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.171351 8987 raft_consensus.cc:374] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:34.171378 8987 raft_consensus.cc:729] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f50f21c8f849eeb01f9ed22733ac91, State: Initialized, Role: FOLLOWER
I20260923 22:17:34.171425 8987 consensus_queue.cc:260] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.171629 8987 ts_tablet_manager.cc:1428] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:34.171691 8989 tablet_bootstrap.cc:492] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: No bootstrap required, opened a new log
I20260923 22:17:34.171736 8989 ts_tablet_manager.cc:1397] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:34.171900 8988 tablet_bootstrap.cc:492] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: No bootstrap required, opened a new log
I20260923 22:17:34.171943 8988 ts_tablet_manager.cc:1397] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:34.171948 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.171852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.65:59635)
0923 22:17:34.171891 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.171925 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260923 22:17:34.172061 8988 raft_consensus.cc:348] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.172122 8988 raft_consensus.cc:374] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:34.172158 8988 raft_consensus.cc:729] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c59dfe33c347d690d5f0ce962522f6, State: Initialized, Role: FOLLOWER
I20260923 22:17:34.171859 8989 raft_consensus.cc:348] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.172231 8989 raft_consensus.cc:374] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:34.172219 8988 consensus_queue.cc:260] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.172257 8989 raft_consensus.cc:729] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7571de4fee408e94e2d7001258bd55, State: Initialized, Role: FOLLOWER
I20260923 22:17:34.172312 8989 consensus_queue.cc:260] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.172389 8988 ts_tablet_manager.cc:1428] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:34.172489 8989 ts_tablet_manager.cc:1428] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:34.172736 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.172584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:52445)
0923 22:17:34.172676 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.172704 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 22:17:34.172835 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.172746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:52175)
0923 22:17:34.172783 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.172807 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 22:17:34.173187 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.173112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:34419)
0923 22:17:34.173145 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.173169 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260923 22:17:34.173650 8888 heartbeater.cc:503] Master 127.1.129.125:45633 was elected leader, sending a full tablet report...
I20260923 22:17:34.173971 8752 heartbeater.cc:503] Master 127.1.129.125:45633 was elected leader, sending a full tablet report...
I20260923 22:17:34.174098 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.173914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:49041)
0923 22:17:34.174040 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.174067 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 22:17:34.234283 8993 raft_consensus.cc:482] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:34.234404 8993 raft_consensus.cc:504] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.234808 8993 leader_election.cc:288] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7c59dfe33c347d690d5f0ce962522f6 (127.1.129.67:36269), 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403)
I20260923 22:17:34.238137 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.234966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:36269 (local address 127.1.129.65:46001)
0923 22:17:34.235259 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.235268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:34.235322 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:34.235452 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:34.235453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:34.235473 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:34.235541 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.235543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.236264 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.236265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.237170 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.237173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.237861 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.237862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.237937 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.237941 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:34.237992 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:17:34.238031 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":71,"thread_start_us":136,"threads_started":1}
I20260923 22:17:34.238157 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.235138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:46001 (local address 127.1.129.67:36269)
0923 22:17:34.235189 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.235192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.235332 (+ 140us) server_negotiation.cc:407] Connection header received
0923 22:17:34.235368 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.235369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.235380 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.235452 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:34.235609 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.236171 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.237299 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.237769 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.237807 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.237814 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.238039 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.238088 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 22:17:34.238209 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.235120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:53735 (local address 127.1.129.66:39403)
0923 22:17:34.235183 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.235185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.235197 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:34.235229 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.235230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.235239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.235309 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:34.235499 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.236082 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.237272 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.237784 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.237924 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.237935 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.238088 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.238125 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:17:34.238219 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.234966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:39403 (local address 127.1.129.65:53735)
0923 22:17:34.235065 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.235079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:34.235158 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:34.235342 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:34.235343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:34.235347 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:34.235420 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.235423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.236189 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.236190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.237192 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.237195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.237966 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.237967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.238028 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.238030 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:34.238080 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:17:34.238110 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":44}
I20260923 22:17:34.238476 8704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315b26925391427b8abae6cba5fe351b" candidate_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7571de4fee408e94e2d7001258bd55" is_pre_election: true
I20260923 22:17:34.238474 8840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315b26925391427b8abae6cba5fe351b" candidate_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c59dfe33c347d690d5f0ce962522f6" is_pre_election: true
I20260923 22:17:34.238585 8704 raft_consensus.cc:2454] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f50f21c8f849eeb01f9ed22733ac91 in term 0.
I20260923 22:17:34.238598 8840 raft_consensus.cc:2454] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f50f21c8f849eeb01f9ed22733ac91 in term 0.
I20260923 22:17:34.238813 8502 leader_election.cc:302] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7571de4fee408e94e2d7001258bd55, d1f50f21c8f849eeb01f9ed22733ac91; no voters:
I20260923 22:17:34.238948 8993 raft_consensus.cc:2781] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:34.239001 8993 raft_consensus.cc:482] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:34.239037 8993 raft_consensus.cc:3037] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:34.239809 8993 raft_consensus.cc:504] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.239974 8993 leader_election.cc:288] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [CANDIDATE]: Term 1 election: Requested vote from peers c7c59dfe33c347d690d5f0ce962522f6 (127.1.129.67:36269), 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403)
I20260923 22:17:34.240173 8840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315b26925391427b8abae6cba5fe351b" candidate_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c59dfe33c347d690d5f0ce962522f6"
I20260923 22:17:34.240155 8704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315b26925391427b8abae6cba5fe351b" candidate_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7571de4fee408e94e2d7001258bd55"
I20260923 22:17:34.240245 8704 raft_consensus.cc:3037] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:34.240245 8840 raft_consensus.cc:3037] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:34.240965 8704 raft_consensus.cc:2454] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f50f21c8f849eeb01f9ed22733ac91 in term 1.
I20260923 22:17:34.241075 8840 raft_consensus.cc:2454] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f50f21c8f849eeb01f9ed22733ac91 in term 1.
I20260923 22:17:34.241205 8502 leader_election.cc:302] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7571de4fee408e94e2d7001258bd55, d1f50f21c8f849eeb01f9ed22733ac91; no voters:
I20260923 22:17:34.241320 8993 raft_consensus.cc:2781] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:34.241508 8993 raft_consensus.cc:686] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [term 1 LEADER]: Becoming Leader. State: Replica: d1f50f21c8f849eeb01f9ed22733ac91, State: Running, Role: LEADER
I20260923 22:17:34.241595 8993 consensus_queue.cc:237] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } }
I20260923 22:17:34.241978 8615 heartbeater.cc:503] Master 127.1.129.125:45633 was elected leader, sending a full tablet report...
I20260923 22:17:34.242230 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.242082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.65:52533)
0923 22:17:34.242166 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.242195 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 22:17:34.242307 8247 catalog_manager.cc:5495] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f50f21c8f849eeb01f9ed22733ac91 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f50f21c8f849eeb01f9ed22733ac91" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:34.261438 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.258287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.1.129.65:32841)
0923 22:17:34.258339 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.258340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.258352 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:34.258459 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.258459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.258476 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.258538 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:34.258735 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.259350 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.260596 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.260820 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.260914 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.261062 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:34.261184 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:34.261278 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.261322 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.261346 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:34.262619 8704 raft_consensus.cc:1261] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 1 FOLLOWER]: Refusing update from remote peer d1f50f21c8f849eeb01f9ed22733ac91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:34.262755 8840 raft_consensus.cc:1261] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 1 FOLLOWER]: Refusing update from remote peer d1f50f21c8f849eeb01f9ed22733ac91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:34.263002 8997 consensus_queue.cc:1035] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7571de4fee408e94e2d7001258bd55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:34.263144 8993 consensus_queue.cc:1035] T 315b26925391427b8abae6cba5fe351b P d1f50f21c8f849eeb01f9ed22733ac91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c59dfe33c347d690d5f0ce962522f6" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:34.264045 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.263928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:57443)
0923 22:17:34.263978 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.264008 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:17:34.264213 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.264036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:51319)
0923 22:17:34.264152 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.264180 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260923 22:17:34.265475 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.265350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.65:48725)
0923 22:17:34.265411 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.265442 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 22:17:34.265727 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.265436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.67:44957)
0923 22:17:34.265653 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.265684 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260923 22:17:34.266207 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.266033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37359 (local address 127.1.129.66:50113)
0923 22:17:34.266148 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.266177 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260923 22:17:34.267449 8247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:34.267552 8247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:34.268842 8247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a0956b8965348e587ec5485f0ce8064: Sending DeleteTablet for 3 replicas of tablet 315b26925391427b8abae6cba5fe351b
I20260923 22:17:34.269140 8820 tablet_service.cc:1514] Processing DeleteTablet for tablet 315b26925391427b8abae6cba5fe351b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:34 UTC) from {username='oryx', principal='oryx/127.1.129.125@KRBTEST.COM'} at 127.0.0.1:53840
I20260923 22:17:34.269155 8547 tablet_service.cc:1514] Processing DeleteTablet for tablet 315b26925391427b8abae6cba5fe351b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:34 UTC) from {username='oryx', principal='oryx/127.1.129.125@KRBTEST.COM'} at 127.0.0.1:58222
I20260923 22:17:34.269336 8684 tablet_service.cc:1514] Processing DeleteTablet for tablet 315b26925391427b8abae6cba5fe351b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:34 UTC) from {username='oryx', principal='oryx/127.1.129.125@KRBTEST.COM'} at 127.0.0.1:52644
I20260923 22:17:34.269526 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 8395
I20260923 22:17:34.269760 9012 tablet_replica.cc:330] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: stopping tablet replica
I20260923 22:17:34.269814 9012 raft_consensus.cc:2229] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:34.269853 9012 raft_consensus.cc:2258] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:34.270200 9011 tablet_replica.cc:330] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: stopping tablet replica
I20260923 22:17:34.270255 9011 raft_consensus.cc:2229] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:34.270305 9011 raft_consensus.cc:2258] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:34.270367 9012 ts_tablet_manager.cc:1905] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:34.270614 9011 ts_tablet_manager.cc:1905] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:34.272089 9012 ts_tablet_manager.cc:1918] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:34.272163 9012 log.cc:1196] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/315b26925391427b8abae6cba5fe351b
I20260923 22:17:34.272327 9011 ts_tablet_manager.cc:1918] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:34.272394 9011 log.cc:1196] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/315b26925391427b8abae6cba5fe351b
I20260923 22:17:34.272437 9012 ts_tablet_manager.cc:1939] T 315b26925391427b8abae6cba5fe351b P 3d7571de4fee408e94e2d7001258bd55: Deleting consensus metadata
I20260923 22:17:34.272661 9011 ts_tablet_manager.cc:1939] T 315b26925391427b8abae6cba5fe351b P c7c59dfe33c347d690d5f0ce962522f6: Deleting consensus metadata
I20260923 22:17:34.272833 8233 catalog_manager.cc:4841] TS 3d7571de4fee408e94e2d7001258bd55 (127.1.129.66:39403): tablet 315b26925391427b8abae6cba5fe351b (table test-table [id=b1218dbf8bc846bd866c83987c7fffb3]) successfully deleted
I20260923 22:17:34.273016 8232 catalog_manager.cc:4841] TS c7c59dfe33c347d690d5f0ce962522f6 (127.1.129.67:36269): tablet 315b26925391427b8abae6cba5fe351b (table test-table [id=b1218dbf8bc846bd866c83987c7fffb3]) successfully deleted
I20260923 22:17:34.276185 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 8621
I20260923 22:17:34.276201 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.276072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:32841 (local address 127.0.0.1:58244)
0923 22:17:34.276124 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.276149 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 22:17:34.276361 8234 catalog_manager.cc:4584] TS d1f50f21c8f849eeb01f9ed22733ac91 (127.1.129.65:32841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 315b26925391427b8abae6cba5fe351b: Network error: Client connection negotiation failed: client connection to 127.1.129.65:32841: connect: Connection refused (error 111)
I20260923 22:17:34.281116 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 8757
I20260923 22:17:34.286523 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 8113
I20260923 22:17:34.291272 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 8301
2026-09-23T22:17:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6174 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:34 dist-test-slave-dmw7 krb5kdc[9015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:34 dist-test-slave-dmw7 krb5kdc[9015](info): set up 2 sockets
Sep 23 22:17:34 dist-test-slave-dmw7 krb5kdc[9015](info): commencing operation
krb5kdc: starting...
W20260923 22:17:35.343130 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:35 dist-test-slave-dmw7 krb5kdc[9015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.126.keytab.
Entry for principal oryx/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.126.keytab.
I20260923 22:17:35.462236 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:41621
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.126.keytab
--principal=oryx/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:35.540632 9028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:35.540846 9028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:35.540879 9028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:35.542069 9028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:35.542117 9028 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 22:17:35.542131 9028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:35.542145 9028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:35.542157 9028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:35.542169 9028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:35.543738 9028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:41621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.126.keytab
--principal=oryx/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:35.544003 9028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:35.544256 9028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:35.549077 9034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.549077 9033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.549105 9036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:35.549260 9028 server_base.cc:1034] running on GCE node
I20260923 22:17:35.549580 9028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:35.549880 9028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:35.551080 9028 hybrid_clock.cc:648] HybridClock initialized: now 1790201855551068 us; error 29 us; skew 500 ppm
Sep 23 22:17:35 dist-test-slave-dmw7 krb5kdc[9015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:35.553632 9028 init.cc:377] Logged in from keytab as oryx/127.1.129.126@KRBTEST.COM (short username oryx)
I20260923 22:17:35.554077 9028 webserver.cc:460] Webserver started at http://127.1.129.126:35565/ using document root <none> and password file <none>
I20260923 22:17:35.554297 9028 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:35.554340 9028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:35.554510 9028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:35.555581 9028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "88658f5a52c740308540176a0f2d7274"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "942c73fd9c90b3e35fa80b39f5d92ea4"
server_key_iv: "eb0834e12de8bc17fdf26bcddab511f6"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.555938 9028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "88658f5a52c740308540176a0f2d7274"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "942c73fd9c90b3e35fa80b39f5d92ea4"
server_key_iv: "eb0834e12de8bc17fdf26bcddab511f6"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.557217 9028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:35.557960 9043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.558153 9028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:35.558243 9028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "88658f5a52c740308540176a0f2d7274"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "942c73fd9c90b3e35fa80b39f5d92ea4"
server_key_iv: "eb0834e12de8bc17fdf26bcddab511f6"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.558298 9028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:35.569339 9028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:35.571169 9028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:35.571277 9028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:35.575503 9028 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:41621
I20260923 22:17:35.575578 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:41621 every 8 connection(s)
I20260923 22:17:35.575901 9028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:35.576543 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 9028
I20260923 22:17:35.576618 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:35.576611 9106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:35.576807 1541 external_mini_cluster.cc:1428] Setting key be0659d7b6ba99c975822113dff3048e
I20260923 22:17:35.579862 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: Bootstrap starting.
I20260923 22:17:35.580526 9106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:35.580816 9106 log.cc:824] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:35.581494 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: No bootstrap required, opened a new log
Sep 23 22:17:35 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:35.583904 9106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88658f5a52c740308540176a0f2d7274" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 41621 } }
I20260923 22:17:35.584036 9106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:35.584059 9106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88658f5a52c740308540176a0f2d7274, State: Initialized, Role: FOLLOWER
I20260923 22:17:35.584198 9106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88658f5a52c740308540176a0f2d7274" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 41621 } }
I20260923 22:17:35.584259 9106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:35.584280 9106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:35.584347 9106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:35.585040 9106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88658f5a52c740308540176a0f2d7274" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 41621 } }
I20260923 22:17:35.585150 9106 leader_election.cc:302] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88658f5a52c740308540176a0f2d7274; no voters:
I20260923 22:17:35.585348 9106 leader_election.cc:288] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:35.585393 9111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:35.585520 9111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [term 1 LEADER]: Becoming Leader. State: Replica: 88658f5a52c740308540176a0f2d7274, State: Running, Role: LEADER
I20260923 22:17:35.585608 9106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:35.585701 9111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88658f5a52c740308540176a0f2d7274" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 41621 } }
I20260923 22:17:35.586089 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88658f5a52c740308540176a0f2d7274. Latest consensus state: current_term: 1 leader_uuid: "88658f5a52c740308540176a0f2d7274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88658f5a52c740308540176a0f2d7274" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 41621 } } }
I20260923 22:17:35.586190 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88658f5a52c740308540176a0f2d7274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88658f5a52c740308540176a0f2d7274" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 41621 } } }
I20260923 22:17:35.586246 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:35.586278 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:35.586493 9120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:35.587368 9120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:35.588071 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.577342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.1.129.126:41621)
0923 22:17:35.577542 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:35.577551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:35.577581 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:17:35.578096 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:35.578104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:35.578165 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:35.578313 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:35.578952 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.580302 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.581116 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.581514 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.581653 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.584186 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:35.584190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:35.584190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:35.584202 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:35.586189 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.586622 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.586625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.586625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.586749 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.586970 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.586971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.586971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.587274 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:35.587416 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:35.587524 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 22:17:35.587763 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260923 22:17:35.589046 9120 catalog_manager.cc:1347] Generated new cluster ID: 9d8d797fdddb475188053aea31856069
I20260923 22:17:35.589095 9120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:35.631052 9120 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:35.631610 9120 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:35.639555 9120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: Generated new TSK 0
I20260923 22:17:35.639781 9120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab.
Entry for principal oryx/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab.
I20260923 22:17:35.673557 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:41621
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab
--principal=oryx/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:35.754575 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:35.754813 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:35.754839 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:35.755987 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:35.756031 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:35.756052 9132 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:35.756071 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:35.757552 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab
--principal=oryx/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:41621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:35.757826 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:35.758023 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:35.763283 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.763283 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.763360 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:35.763653 9132 server_base.cc:1034] running on GCE node
I20260923 22:17:35.763923 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:35.764235 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:35.765445 9132 hybrid_clock.cc:648] HybridClock initialized: now 1790201855765427 us; error 32 us; skew 500 ppm
Sep 23 22:17:35 dist-test-slave-dmw7 krb5kdc[9015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:35.768298 9132 init.cc:377] Logged in from keytab as oryx/127.1.129.65@KRBTEST.COM (short username oryx)
I20260923 22:17:35.768743 9132 webserver.cc:460] Webserver started at http://127.1.129.65:43191/ using document root <none> and password file <none>
I20260923 22:17:35.768982 9132 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:35.769030 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:35.769202 9132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:35.770293 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "c5820834132c4546a6efb1374e6f3bc9"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "6ef83e4799899405c9119c2fd95ca935"
server_key_iv: "cf1f30be4782ddffc4e5cb11809b3428"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.770717 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "c5820834132c4546a6efb1374e6f3bc9"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "6ef83e4799899405c9119c2fd95ca935"
server_key_iv: "cf1f30be4782ddffc4e5cb11809b3428"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.772131 9132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:35.772918 9147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.773115 9132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:35.773212 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "c5820834132c4546a6efb1374e6f3bc9"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "6ef83e4799899405c9119c2fd95ca935"
server_key_iv: "cf1f30be4782ddffc4e5cb11809b3428"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.773293 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:35.781702 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:35.784009 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:35.784245 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:35.784834 9132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:35.784875 9132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.784900 9132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:35.784950 9132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 22:17:35 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:35.792680 9132 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:36563
I20260923 22:17:35.793004 9262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:36563 every 8 connection(s)
I20260923 22:17:35.793049 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:35.795341 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.785325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:45827 (local address 127.1.129.126:41621)
0923 22:17:35.785430 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:35.785433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:35.785776 (+ 343us) server_negotiation.cc:407] Connection header received
0923 22:17:35.786351 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:35.786352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:35.786368 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:35.786465 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:35.787186 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.787967 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.789488 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.789848 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.789981 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.792442 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:35.792443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:35.792444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:35.792451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:35.794089 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.794455 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.794455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.794456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.794513 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.794781 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.794782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.794782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.794890 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:35.794949 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:35.795208 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 22:17:35.795278 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 22:17:35.795414 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.785319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:41621 (local address 127.1.129.65:45827)
0923 22:17:35.785634 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:35.785674 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:35.786155 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:35.786537 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:35.786542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:35.786884 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:35.787075 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.787080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.788083 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.788084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.789391 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.789394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.790091 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.790092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.790236 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.790776 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:35.790789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:35.792333 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:35.794169 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.794173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.794175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.794384 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.794568 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.794568 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.794568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.794725 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.794955 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:35.794959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:35.795066 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 22:17:35.795142 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 22:17:35.795253 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":251,"threads_started":1}
I20260923 22:17:35.795940 9263 heartbeater.cc:347] Connected to a master server at 127.1.129.126:41621
I20260923 22:17:35.796025 9263 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:35.796227 9263 heartbeater.cc:511] Master 127.1.129.126:41621 requested a full tablet report, sending...
I20260923 22:17:35.796751 9059 ts_manager.cc:194] Registered new tserver with Master: c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65:36563)
I20260923 22:17:35.797626 9059 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:45827
I20260923 22:17:35.798403 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 9132
I20260923 22:17:35.798501 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:35.798712 1541 external_mini_cluster.cc:1428] Setting key 44d2146db3a3be2fe33bb605f376831f
I20260923 22:17:35.803248 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.796989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:41621 (local address 127.1.129.65:44739)
0923 22:17:35.797104 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:35.797114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:35.797172 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:35.797298 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:35.797299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:35.797430 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:35.797490 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.797491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.798302 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.798303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.799106 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.799109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.799486 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.799487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.799554 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.799951 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:35.799956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:35.800375 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:35.802277 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.802278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.802279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.802436 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.802676 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.802677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.802677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.802740 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.803013 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:35.803014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:35.803060 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 22:17:35.803111 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:17:35.803182 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260923 22:17:35.803376 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.797055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:44739 (local address 127.1.129.126:41621)
0923 22:17:35.797098 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:35.797100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:35.797158 (+ 58us) server_negotiation.cc:407] Connection header received
0923 22:17:35.797212 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:35.797213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:35.797223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:35.797298 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:35.797556 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.798191 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.799185 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.799404 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.799473 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.800544 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:35.800545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:35.800545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:35.800551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:35.802196 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.802504 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.802505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.802506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.802563 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.802806 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.802807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.802807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.802946 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:35.803015 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:35.803156 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 22:17:35.803248 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for oryx/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab.
Entry for principal oryx/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab.
I20260923 22:17:35.825071 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:41621
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab
--principal=oryx/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:35.908109 9272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:35.908377 9272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:35.908417 9272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:35.909564 9272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:35.909643 9272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:35.909677 9272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:35.909721 9272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:35.911283 9272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab
--principal=oryx/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:41621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:35.911573 9272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:35.911816 9272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:35.917394 9280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.917385 9277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.917385 9278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:35.917615 9272 server_base.cc:1034] running on GCE node
I20260923 22:17:35.917928 9272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:35.918231 9272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:35.919391 9272 hybrid_clock.cc:648] HybridClock initialized: now 1790201855919379 us; error 29 us; skew 500 ppm
Sep 23 22:17:35 dist-test-slave-dmw7 krb5kdc[9015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:35.922083 9272 init.cc:377] Logged in from keytab as oryx/127.1.129.66@KRBTEST.COM (short username oryx)
I20260923 22:17:35.922737 9272 webserver.cc:460] Webserver started at http://127.1.129.66:33339/ using document root <none> and password file <none>
I20260923 22:17:35.923025 9272 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:35.923110 9272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:35.923293 9272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:35.924481 9272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "1396b309221441f2b4189cd198b16574"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "5e0eb4688344ae24940f7c5795cbd115"
server_key_iv: "6ecb3035494b97d3d0da585b8ebbced2"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.924913 9272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "1396b309221441f2b4189cd198b16574"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "5e0eb4688344ae24940f7c5795cbd115"
server_key_iv: "6ecb3035494b97d3d0da585b8ebbced2"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.926451 9272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:35.927358 9287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.927565 9272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:35.927683 9272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "1396b309221441f2b4189cd198b16574"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-dmw7"
server_key: "5e0eb4688344ae24940f7c5795cbd115"
server_key_iv: "6ecb3035494b97d3d0da585b8ebbced2"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.927771 9272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:35.947167 9272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:35.948982 9272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:35.949149 9272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:35.949747 9272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:35.949795 9272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.949831 9272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:35.949849 9272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 22:17:35 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:35.957115 9272 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:33741
I20260923 22:17:35.957304 9402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:33741 every 8 connection(s)
I20260923 22:17:35.957549 9272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:35.959595 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.950234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:59365 (local address 127.1.129.126:41621)
0923 22:17:35.950308 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:35.950311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:35.950552 (+ 241us) server_negotiation.cc:407] Connection header received
0923 22:17:35.951357 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:35.951358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:35.951373 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:35.951431 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:35.952067 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.952556 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.953576 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.953771 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.953861 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.956712 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:35.956713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:35.956713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:35.956719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:35.958315 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.958763 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.958764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.958765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.958824 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.959028 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.959028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.959028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.959123 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:35.959167 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:35.959444 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 22:17:35.959521 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 22:17:35.959623 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.950216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:41621 (local address 127.1.129.66:59365)
0923 22:17:35.950405 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:35.950455 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:35.951215 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:35.951524 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:35.951529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:35.951783 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:35.951970 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.951974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.952656 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.952657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.953479 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.953481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.953860 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.953861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.954023 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.954614 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:35.954622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:35.956615 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:35.958391 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.958394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.958396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.958604 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.958884 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.958884 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.958884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.958975 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.959178 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:35.959182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:35.959291 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 22:17:35.959366 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 22:17:35.959478 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
I20260923 22:17:35.959849 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 9272
I20260923 22:17:35.959980 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:35.960175 1541 external_mini_cluster.cc:1428] Setting key 74249e42a96e840ebe25567dbfe1fb3f
I20260923 22:17:35.960222 9403 heartbeater.cc:347] Connected to a master server at 127.1.129.126:41621
I20260923 22:17:35.960296 9403 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:35.960493 9403 heartbeater.cc:511] Master 127.1.129.126:41621 requested a full tablet report, sending...
I20260923 22:17:35.960868 9060 ts_manager.cc:194] Registered new tserver with Master: 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741)
I20260923 22:17:35.961376 9060 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:59365
I20260923 22:17:35.967854 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.961273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:41621 (local address 127.1.129.66:34157)
0923 22:17:35.961395 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:35.961405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:35.961452 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:35.961676 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:35.961677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:35.961873 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:35.961935 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.961937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.962797 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.962798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.963592 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.963594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.964090 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.964091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.964135 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.964589 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:35.964594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:35.964892 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:35.966728 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.966729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.966729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.966902 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.967127 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.967128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.967128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.967185 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.967557 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:35.967558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:35.967601 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 22:17:35.967681 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:17:35.967771 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260923 22:17:35.967911 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.961514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:34157 (local address 127.1.129.126:41621)
0923 22:17:35.961561 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:35.961562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:35.961568 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:17:35.961583 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:35.961583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:35.961591 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:35.961654 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:35.962024 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.962694 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.963676 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.964017 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.964086 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.964980 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:35.964981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:35.964982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:35.964989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:35.966567 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.966987 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.966988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.966988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.967052 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.967253 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.967254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.967254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.967459 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:35.967539 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:35.967720 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 22:17:35.967800 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab.
Entry for principal oryx/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab.
I20260923 22:17:35.987641 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:41621
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab
--principal=oryx/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:36.068836 9412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:36.069042 9412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:36.069065 9412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:36.070165 9412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:36.070202 9412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:36.070222 9412 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:36.070240 9412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:36.071781 9412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab
--principal=oryx/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.129.126:41621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:36.072024 9412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:36.072250 9412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:36.077360 9417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.077436 9420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.077421 9412 server_base.cc:1034] running on GCE node
W20260923 22:17:36.077379 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.077821 9412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:36.078080 9412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:36.079272 9412 hybrid_clock.cc:648] HybridClock initialized: now 1790201856079249 us; error 37 us; skew 500 ppm
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:36.082062 9412 init.cc:377] Logged in from keytab as oryx/127.1.129.67@KRBTEST.COM (short username oryx)
I20260923 22:17:36.082549 9412 webserver.cc:460] Webserver started at http://127.1.129.67:44959/ using document root <none> and password file <none>
I20260923 22:17:36.082813 9412 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.082891 9412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.083045 9412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:36.084120 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "a66ea7ad8fe64bdcb4a852e9395ea150"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-dmw7"
server_key: "fc4055748f65e9c311dd0b6376e7c1e4"
server_key_iv: "68c02c3fceeb7323ea8214b7ca34a9a0"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.084513 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "a66ea7ad8fe64bdcb4a852e9395ea150"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-dmw7"
server_key: "fc4055748f65e9c311dd0b6376e7c1e4"
server_key_iv: "68c02c3fceeb7323ea8214b7ca34a9a0"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.085909 9412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:36.086818 9427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.086998 9412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.087098 9412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "a66ea7ad8fe64bdcb4a852e9395ea150"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-dmw7"
server_key: "fc4055748f65e9c311dd0b6376e7c1e4"
server_key_iv: "68c02c3fceeb7323ea8214b7ca34a9a0"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.087188 9412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.105971 9412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:36.108455 9412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:36.108628 9412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:36.109143 9412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:36.109211 9412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.109288 9412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:36.109320 9412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:36.116536 9412 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:33745
I20260923 22:17:36.116617 9542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:33745 every 8 connection(s)
I20260923 22:17:36.116926 9412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:36.119369 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.109718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:42079 (local address 127.1.129.126:41621)
0923 22:17:36.109756 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.109758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.110028 (+ 270us) server_negotiation.cc:407] Connection header received
0923 22:17:36.110533 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.110535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.110547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.110604 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.111489 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.112329 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.114014 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.114327 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.114417 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.116707 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.116708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.116708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.116716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.118166 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.118552 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.118553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.118553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.118602 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.118879 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.118880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.118880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.118983 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.119035 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.119245 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.119308 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:17:36.119592 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.109711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:41621 (local address 127.1.129.67:42079)
0923 22:17:36.109903 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.109937 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.110427 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.110701 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.110705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.111120 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.111395 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.111401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.112443 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.112445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.113914 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.113918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.114422 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.114424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.114570 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.115176 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.115186 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.116577 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.118249 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.118254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.118256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.118483 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.118721 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.118721 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.118722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.118818 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.119043 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.119047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.119152 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.119230 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.119342 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260923 22:17:36.120124 9543 heartbeater.cc:347] Connected to a master server at 127.1.129.126:41621
I20260923 22:17:36.120203 9543 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:36.120373 9543 heartbeater.cc:511] Master 127.1.129.126:41621 requested a full tablet report, sending...
I20260923 22:17:36.120750 9059 ts_manager.cc:194] Registered new tserver with Master: a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745)
I20260923 22:17:36.121228 9059 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:42079
I20260923 22:17:36.122474 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 9412
I20260923 22:17:36.122586 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:36.122790 1541 external_mini_cluster.cc:1428] Setting key d66a7f5ea54fc3e93bf721495ccdebce
I20260923 22:17:36.123639 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:36.127111 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.121217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:42709 (local address 127.1.129.126:41621)
0923 22:17:36.121260 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.121261 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.121323 (+ 62us) server_negotiation.cc:407] Connection header received
0923 22:17:36.121396 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.121397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.121407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.121451 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.121772 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.122227 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.123098 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.123347 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.123447 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.124371 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.124372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.124372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.124379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.125988 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.126343 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.126344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.126345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.126401 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.126615 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.126616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.126616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.126770 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.126821 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.126997 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.127053 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:36.127310 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.121144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:41621 (local address 127.1.129.67:42709)
0923 22:17:36.121262 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.121273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.121346 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.121463 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.121464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.121636 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.121687 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.121688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.122328 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.122328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.123003 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.123005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.123449 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.123450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.123513 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.123978 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.123983 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.124252 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.126075 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.126076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.126077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.126261 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.126477 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.126478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.126478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.126547 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.126839 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.126840 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.126893 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.126969 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.127086 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 22:17:36.144996 9109 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:36.139948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.1.129.126:41621)
0923 22:17:36.140003 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.140005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.140031 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:17:36.140117 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.140117 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.140125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.140171 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.140703 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.141165 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.141883 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.142053 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.142101 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.144941 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47314: BlockingRecv error: recv got EOF from 127.0.0.1:47314 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:36.151764 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.145757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.1.129.126:41621)
0923 22:17:36.145825 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.145847 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.145859 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:36.145933 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.145934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.145944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.146010 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.146415 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.146925 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.147539 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.147703 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.147750 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.149672 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.149672 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.149673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.149677 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.150902 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.151183 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.151184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.151184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.151243 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.151392 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.151392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.151393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.151476 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.151539 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.151621 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.151670 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 22:17:36.153398 9059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:36.154955 9059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:36.161484 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.157137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:33745 (local address 127.0.0.1:48400)
0923 22:17:36.157456 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.157496 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.157565 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.157940 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.157940 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.157942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:36.157984 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.157985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.158948 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.158949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.160295 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.160297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.161264 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.161265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.161321 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.161326 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.161376 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.161410 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":65,"thread_start_us":40,"threads_started":1}
I20260923 22:17:36.161494 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.157430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.1.129.65:36563)
0923 22:17:36.157587 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.157591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.157615 (+ 24us) server_negotiation.cc:407] Connection header received
0923 22:17:36.157684 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.157686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.157702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.157781 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:36.157955 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.159137 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.160422 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.160985 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.161084 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.161164 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.161344 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.161392 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260923 22:17:36.161548 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.157305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.1.129.67:33745)
0923 22:17:36.157699 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.157703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.157712 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:36.157774 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.157776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.157792 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.157876 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:36.158056 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.158861 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.160390 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.161137 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.161291 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.161370 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.161408 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.161450 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":95,"threads_started":1}
I20260923 22:17:36.161731 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.157189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:33741 (local address 127.0.0.1:43918)
0923 22:17:36.157490 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.157500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.157564 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.157920 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.157921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.157923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:36.157984 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.157986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.159227 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.159228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.160434 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.160436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.161451 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.161452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.161518 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.161522 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.161609 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.161652 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":53,"thread_start_us":74,"threads_started":1}
I20260923 22:17:36.161898 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.157301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.1.129.66:33741)
0923 22:17:36.157714 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.157718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.157729 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:36.157764 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.157766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.157780 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.157845 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:36.158066 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.159137 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.160537 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.161291 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.161348 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.161419 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.161791 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.161824 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":102,"threads_started":1}
I20260923 22:17:36.161739 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.157316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:36563 (local address 127.0.0.1:41592)
0923 22:17:36.157567 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.157576 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.157609 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.157802 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.157805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.157807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:36.157872 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.157873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.159251 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.159252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.160327 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.160329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.161087 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.161088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.161164 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.161179 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.161249 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.161592 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":63,"thread_start_us":81,"threads_started":1}
I20260923 22:17:36.162665 9477 tablet_service.cc:1467] Processing CreateTablet for tablet 05021976a6e947fcb345adf2874b5699 (DEFAULT_TABLE table=test-table [id=157c971d0b514798889904973ce47e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.162811 9337 tablet_service.cc:1467] Processing CreateTablet for tablet 05021976a6e947fcb345adf2874b5699 (DEFAULT_TABLE table=test-table [id=157c971d0b514798889904973ce47e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.162784 9197 tablet_service.cc:1467] Processing CreateTablet for tablet 05021976a6e947fcb345adf2874b5699 (DEFAULT_TABLE table=test-table [id=157c971d0b514798889904973ce47e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.163045 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05021976a6e947fcb345adf2874b5699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.163120 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05021976a6e947fcb345adf2874b5699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.163182 9197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05021976a6e947fcb345adf2874b5699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.166193 9571 tablet_bootstrap.cc:492] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Bootstrap starting.
I20260923 22:17:36.166613 9572 tablet_bootstrap.cc:492] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Bootstrap starting.
I20260923 22:17:36.166986 9571 tablet_bootstrap.cc:654] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.167016 9573 tablet_bootstrap.cc:492] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Bootstrap starting.
I20260923 22:17:36.167336 9572 tablet_bootstrap.cc:654] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.167352 9571 log.cc:824] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:36.167647 9573 tablet_bootstrap.cc:654] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.167650 9572 log.cc:824] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:36.168020 9573 log.cc:824] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:36.168146 9571 tablet_bootstrap.cc:492] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: No bootstrap required, opened a new log
I20260923 22:17:36.168220 9571 ts_tablet_manager.cc:1397] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:36.168406 9572 tablet_bootstrap.cc:492] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: No bootstrap required, opened a new log
I20260923 22:17:36.168473 9572 ts_tablet_manager.cc:1397] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:17:36.170364 9571 raft_consensus.cc:348] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.170365 9572 raft_consensus.cc:348] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.170542 9571 raft_consensus.cc:374] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.170542 9572 raft_consensus.cc:374] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.170576 9571 raft_consensus.cc:729] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.170576 9572 raft_consensus.cc:729] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ea7ad8fe64bdcb4a852e9395ea150, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.170735 9571 consensus_queue.cc:260] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.170735 9572 consensus_queue.cc:260] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.171010 9572 ts_tablet_manager.cc:1428] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 22:17:36.171010 9571 ts_tablet_manager.cc:1428] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:36.171129 9543 heartbeater.cc:503] Master 127.1.129.126:41621 was elected leader, sending a full tablet report...
I20260923 22:17:36.171129 9263 heartbeater.cc:503] Master 127.1.129.126:41621 was elected leader, sending a full tablet report...
I20260923 22:17:36.171216 9573 tablet_bootstrap.cc:492] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: No bootstrap required, opened a new log
I20260923 22:17:36.171280 9573 ts_tablet_manager.cc:1397] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 22:17:36.173298 9573 raft_consensus.cc:348] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.173456 9573 raft_consensus.cc:374] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.173501 9573 raft_consensus.cc:729] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1396b309221441f2b4189cd198b16574, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.173610 9573 consensus_queue.cc:260] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.173870 9573 ts_tablet_manager.cc:1428] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:36.173933 9403 heartbeater.cc:503] Master 127.1.129.126:41621 was elected leader, sending a full tablet report...
W20260923 22:17:36.208160 9404 tablet.cc:2369] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:36.293906 9264 tablet.cc:2369] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:36.367851 9544 tablet.cc:2369] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:36.412660 9577 raft_consensus.cc:482] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:36.412793 9577 raft_consensus.cc:504] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.413147 9577 leader_election.cc:288] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745), 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741)
I20260923 22:17:36.416244 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.413439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:44237 (local address 127.1.129.67:33745)
0923 22:17:36.413490 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.413492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.413593 (+ 101us) server_negotiation.cc:407] Connection header received
0923 22:17:36.413706 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.413707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.413725 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.413811 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:36.413970 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.414423 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.415492 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.415869 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.415974 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.415982 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.416138 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.416173 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:17:36.416242 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.413535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:44045 (local address 127.1.129.66:33741)
0923 22:17:36.413617 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.413619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.413632 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:17:36.413674 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.413675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.413684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.413758 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:36.413935 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.414410 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.415497 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.415870 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.415957 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.415966 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.416121 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.416156 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 22:17:36.416290 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.413380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:33745 (local address 127.1.129.65:44237)
0923 22:17:36.413520 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.413532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.413627 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.413817 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.413817 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.413819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:36.413888 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.413890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.414518 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.414519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.415395 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.415397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.415975 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.415975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.416046 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.416049 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.416131 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.416173 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260923 22:17:36.416308 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.413416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:33741 (local address 127.1.129.65:44045)
0923 22:17:36.413540 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.413550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.413606 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.413765 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.413766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.413768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:36.413857 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.413860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.414500 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.414501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.415397 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.415399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.415966 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.415967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.416037 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.416042 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.416116 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.416161 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":109,"thread_start_us":60,"threads_started":1}
I20260923 22:17:36.416599 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05021976a6e947fcb345adf2874b5699" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" is_pre_election: true
I20260923 22:17:36.416620 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05021976a6e947fcb345adf2874b5699" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574" is_pre_election: true
I20260923 22:17:36.416760 9357 raft_consensus.cc:2454] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 0.
I20260923 22:17:36.416761 9497 raft_consensus.cc:2454] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 0.
I20260923 22:17:36.417004 9150 leader_election.cc:302] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1396b309221441f2b4189cd198b16574, c5820834132c4546a6efb1374e6f3bc9; no voters:
I20260923 22:17:36.417126 9577 raft_consensus.cc:2781] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:36.417194 9577 raft_consensus.cc:482] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:36.417214 9577 raft_consensus.cc:3037] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.417856 9577 raft_consensus.cc:504] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.418061 9577 leader_election.cc:288] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 election: Requested vote from peers a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745), 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741)
I20260923 22:17:36.418133 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05021976a6e947fcb345adf2874b5699" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574"
I20260923 22:17:36.418200 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05021976a6e947fcb345adf2874b5699" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150"
I20260923 22:17:36.418222 9357 raft_consensus.cc:3037] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.418313 9497 raft_consensus.cc:3037] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.419000 9357 raft_consensus.cc:2454] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 1.
I20260923 22:17:36.419086 9497 raft_consensus.cc:2454] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 1.
I20260923 22:17:36.419162 9150 leader_election.cc:302] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1396b309221441f2b4189cd198b16574, c5820834132c4546a6efb1374e6f3bc9; no voters:
I20260923 22:17:36.419260 9577 raft_consensus.cc:2781] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:36.419386 9577 raft_consensus.cc:686] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 1 LEADER]: Becoming Leader. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Running, Role: LEADER
I20260923 22:17:36.419518 9577 consensus_queue.cc:237] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } }
I20260923 22:17:36.420400 9060 catalog_manager.cc:5495] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "c5820834132c4546a6efb1374e6f3bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:36.444028 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.441779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.1.129.65:36563)
0923 22:17:36.441844 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.441846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.441855 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:36.441891 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.441892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.441908 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.441968 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:36.442115 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.442579 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.443339 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.443515 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.443567 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.443745 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:36.443845 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:36.443917 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.443952 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.443971 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:36.446173 9357 raft_consensus.cc:1261] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Refusing update from remote peer c5820834132c4546a6efb1374e6f3bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:36.446173 9497 raft_consensus.cc:1261] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Refusing update from remote peer c5820834132c4546a6efb1374e6f3bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:36.446532 9577 consensus_queue.cc:1035] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:36.446730 9582 consensus_queue.cc:1035] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:36.449227 9587 mvcc.cc:205] Tried to move back new op lower bound from 7332666804000411648 to 7332666803895193600. Current Snapshot: MvccSnapshot[applied={T|T < 7332666804000411648}]
I20260923 22:17:36.449707 9586 mvcc.cc:205] Tried to move back new op lower bound from 7332666804000411648 to 7332666803895193600. Current Snapshot: MvccSnapshot[applied={T|T < 7332666804000411648}]
I20260923 22:17:36.451262 9060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:36.451366 9060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:36.452102 9060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: Sending DeleteTablet for 3 replicas of tablet 05021976a6e947fcb345adf2874b5699
I20260923 22:17:36.452445 9477 tablet_service.cc:1514] Processing DeleteTablet for tablet 05021976a6e947fcb345adf2874b5699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:48400
I20260923 22:17:36.452445 9337 tablet_service.cc:1514] Processing DeleteTablet for tablet 05021976a6e947fcb345adf2874b5699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:43918
I20260923 22:17:36.452485 9197 tablet_service.cc:1514] Processing DeleteTablet for tablet 05021976a6e947fcb345adf2874b5699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:41592
I20260923 22:17:36.452767 9060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:36.452916 9060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:36.452970 9595 tablet_replica.cc:330] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: stopping tablet replica
I20260923 22:17:36.453092 9595 raft_consensus.cc:2229] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:36.453279 9595 raft_consensus.cc:2258] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.453704 9596 tablet_replica.cc:330] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: stopping tablet replica
I20260923 22:17:36.454083 9595 ts_tablet_manager.cc:1905] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.454180 9477 tablet_service.cc:1467] Processing CreateTablet for tablet 49232b49b8944477b494e64e513dc90c (DEFAULT_TABLE table=test-table [id=798614332d2048a596cd4de90875a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.454300 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49232b49b8944477b494e64e513dc90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.454402 9197 tablet_service.cc:1467] Processing CreateTablet for tablet 49232b49b8944477b494e64e513dc90c (DEFAULT_TABLE table=test-table [id=798614332d2048a596cd4de90875a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.454522 9197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49232b49b8944477b494e64e513dc90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.455704 9571 tablet_bootstrap.cc:492] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: Bootstrap starting.
I20260923 22:17:36.455744 9572 tablet_bootstrap.cc:492] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: Bootstrap starting.
I20260923 22:17:36.456058 9571 tablet_bootstrap.cc:654] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.456120 9572 tablet_bootstrap.cc:654] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.456616 9571 tablet_bootstrap.cc:492] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: No bootstrap required, opened a new log
I20260923 22:17:36.456683 9571 ts_tablet_manager.cc:1397] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.456830 9571 raft_consensus.cc:348] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.456874 9571 raft_consensus.cc:374] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.456887 9571 raft_consensus.cc:729] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.456956 9571 consensus_queue.cc:260] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.457054 9571 ts_tablet_manager.cc:1428] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.457679 9572 tablet_bootstrap.cc:492] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: No bootstrap required, opened a new log
I20260923 22:17:36.457748 9572 ts_tablet_manager.cc:1397] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260923 22:17:36.457772 9597 tablet_replica.cc:330] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: stopping tablet replica
I20260923 22:17:36.457863 9597 raft_consensus.cc:2229] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:36.457891 9572 raft_consensus.cc:348] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.457950 9595 ts_tablet_manager.cc:1918] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:36.457962 9597 raft_consensus.cc:2258] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.457973 9572 raft_consensus.cc:374] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.457999 9572 raft_consensus.cc:729] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ea7ad8fe64bdcb4a852e9395ea150, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.458004 9595 log.cc:1196] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/05021976a6e947fcb345adf2874b5699
I20260923 22:17:36.454128 9337 tablet_service.cc:1467] Processing CreateTablet for tablet 49232b49b8944477b494e64e513dc90c (DEFAULT_TABLE table=test-table [id=798614332d2048a596cd4de90875a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.458062 9572 consensus_queue.cc:260] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.458151 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49232b49b8944477b494e64e513dc90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.458163 9572 ts_tablet_manager.cc:1428] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.458317 9595 ts_tablet_manager.cc:1939] T 05021976a6e947fcb345adf2874b5699 P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting consensus metadata
I20260923 22:17:36.458683 9047 catalog_manager.cc:4841] TS a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745): tablet 05021976a6e947fcb345adf2874b5699 (table test-table [id=157c971d0b514798889904973ce47e04]) successfully deleted
I20260923 22:17:36.459517 9573 tablet_bootstrap.cc:492] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: Bootstrap starting.
I20260923 22:17:36.459965 9573 tablet_bootstrap.cc:654] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.460027 9596 raft_consensus.cc:2229] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:36.460132 9596 raft_consensus.cc:2258] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.460420 9597 ts_tablet_manager.cc:1905] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.460661 9573 tablet_bootstrap.cc:492] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: No bootstrap required, opened a new log
I20260923 22:17:36.460736 9573 ts_tablet_manager.cc:1397] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.460852 9573 raft_consensus.cc:348] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.460917 9573 raft_consensus.cc:374] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.460935 9573 raft_consensus.cc:729] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1396b309221441f2b4189cd198b16574, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.461000 9573 consensus_queue.cc:260] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.461089 9573 ts_tablet_manager.cc:1428] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.461416 9596 ts_tablet_manager.cc:1905] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.462785 9596 ts_tablet_manager.cc:1918] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:36.462857 9596 log.cc:1196] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/05021976a6e947fcb345adf2874b5699
I20260923 22:17:36.463099 9596 ts_tablet_manager.cc:1939] T 05021976a6e947fcb345adf2874b5699 P 1396b309221441f2b4189cd198b16574: Deleting consensus metadata
I20260923 22:17:36.463477 9046 catalog_manager.cc:4841] TS 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741): tablet 05021976a6e947fcb345adf2874b5699 (table test-table [id=157c971d0b514798889904973ce47e04]) successfully deleted
I20260923 22:17:36.463613 9597 ts_tablet_manager.cc:1918] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:36.463724 9597 log.cc:1196] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/05021976a6e947fcb345adf2874b5699
I20260923 22:17:36.463997 9597 ts_tablet_manager.cc:1939] T 05021976a6e947fcb345adf2874b5699 P c5820834132c4546a6efb1374e6f3bc9: Deleting consensus metadata
I20260923 22:17:36.464349 9047 catalog_manager.cc:4841] TS c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65:36563): tablet 05021976a6e947fcb345adf2874b5699 (table test-table [id=157c971d0b514798889904973ce47e04]) successfully deleted
I20260923 22:17:36.515393 9582 raft_consensus.cc:482] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:36.515478 9582 raft_consensus.cc:504] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.515635 9582 leader_election.cc:288] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741), a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745)
I20260923 22:17:36.515817 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49232b49b8944477b494e64e513dc90c" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574" is_pre_election: true
I20260923 22:17:36.515853 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49232b49b8944477b494e64e513dc90c" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" is_pre_election: true
I20260923 22:17:36.515944 9357 raft_consensus.cc:2454] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 0.
I20260923 22:17:36.515964 9497 raft_consensus.cc:2454] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 0.
I20260923 22:17:36.516114 9151 leader_election.cc:302] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a66ea7ad8fe64bdcb4a852e9395ea150, c5820834132c4546a6efb1374e6f3bc9; no voters:
I20260923 22:17:36.516245 9582 raft_consensus.cc:2781] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:36.516281 9582 raft_consensus.cc:482] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:36.516311 9582 raft_consensus.cc:3037] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.516968 9582 raft_consensus.cc:504] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.517120 9582 leader_election.cc:288] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741), a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745)
I20260923 22:17:36.517249 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49232b49b8944477b494e64e513dc90c" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574"
I20260923 22:17:36.517314 9357 raft_consensus.cc:3037] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.517349 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49232b49b8944477b494e64e513dc90c" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150"
I20260923 22:17:36.517421 9497 raft_consensus.cc:3037] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.517925 9357 raft_consensus.cc:2454] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 1.
I20260923 22:17:36.518105 9150 leader_election.cc:302] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1396b309221441f2b4189cd198b16574, c5820834132c4546a6efb1374e6f3bc9; no voters:
I20260923 22:17:36.518167 9497 raft_consensus.cc:2454] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 1.
I20260923 22:17:36.518214 9582 raft_consensus.cc:2781] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:36.518254 9582 raft_consensus.cc:686] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 1 LEADER]: Becoming Leader. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Running, Role: LEADER
I20260923 22:17:36.518308 9582 consensus_queue.cc:237] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.519047 9060 catalog_manager.cc:5495] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "c5820834132c4546a6efb1374e6f3bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } health_report { overall_health: UNKNOWN } } }
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.126@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:36.555226 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.547396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.1.129.126:41621)
0923 22:17:36.547445 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.547447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.547565 (+ 118us) server_negotiation.cc:407] Connection header received
0923 22:17:36.547630 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.547630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.547639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.547688 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.548258 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.548736 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.549736 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.549926 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.549974 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.552196 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.552197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.552198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.552205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.553616 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.554181 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.554182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.554183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.554271 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.554504 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.554506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.554506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.554682 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.554763 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.554949 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.555078 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":26}
I20260923 22:17:36.555282 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.547293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:41621 (local address 127.0.0.1:47332)
0923 22:17:36.547477 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.547490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.547567 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.547716 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.547717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.548125 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.548182 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.548185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.548829 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.548830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.549652 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.549654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.550007 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.550008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.550071 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.550598 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.550604 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.552099 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.553748 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.553756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.553758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.553968 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.554353 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.554354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.554355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.554429 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.554753 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.554761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.554811 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.554920 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.555141 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":104,"thread_start_us":97,"threads_started":1}
I20260923 22:17:36.557400 9060 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:47332:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 22:17:36.557592 9060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:36.559237 9337 tablet_service.cc:1467] Processing CreateTablet for tablet c41a909ea30a4c87937278d5722bf36e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=588246e588394d10a7506096feee9174]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:36.559237 9477 tablet_service.cc:1467] Processing CreateTablet for tablet c41a909ea30a4c87937278d5722bf36e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=588246e588394d10a7506096feee9174]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:36.559414 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a909ea30a4c87937278d5722bf36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.559415 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a909ea30a4c87937278d5722bf36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.560838 9573 tablet_bootstrap.cc:492] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: Bootstrap starting.
I20260923 22:17:36.560950 9197 tablet_service.cc:1467] Processing CreateTablet for tablet c41a909ea30a4c87937278d5722bf36e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=588246e588394d10a7506096feee9174]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:36.561096 9197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a909ea30a4c87937278d5722bf36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.561265 9573 tablet_bootstrap.cc:654] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.562318 9571 tablet_bootstrap.cc:492] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: Bootstrap starting.
I20260923 22:17:36.562417 9572 tablet_bootstrap.cc:492] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: Bootstrap starting.
I20260923 22:17:36.562810 9571 tablet_bootstrap.cc:654] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.562868 9572 tablet_bootstrap.cc:654] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.563460 9573 tablet_bootstrap.cc:492] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: No bootstrap required, opened a new log
I20260923 22:17:36.563519 9573 ts_tablet_manager.cc:1397] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 22:17:36.563488 9571 tablet_bootstrap.cc:492] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: No bootstrap required, opened a new log
I20260923 22:17:36.563565 9571 ts_tablet_manager.cc:1397] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.563669 9573 raft_consensus.cc:348] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.563731 9573 raft_consensus.cc:374] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.563750 9573 raft_consensus.cc:729] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1396b309221441f2b4189cd198b16574, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.563723 9571 raft_consensus.cc:348] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.563773 9571 raft_consensus.cc:374] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.563817 9571 raft_consensus.cc:729] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.563795 9573 consensus_queue.cc:260] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.563864 9571 consensus_queue.cc:260] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.563899 9573 ts_tablet_manager.cc:1428] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.563957 9571 ts_tablet_manager.cc:1428] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.563978 9579 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.564074 9579 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:36.564064 9590 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.564131 9590 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:36.564373 9572 tablet_bootstrap.cc:492] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: No bootstrap required, opened a new log
I20260923 22:17:36.564419 9572 ts_tablet_manager.cc:1397] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:17:36.564563 9572 raft_consensus.cc:348] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.564626 9572 raft_consensus.cc:374] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.564646 9572 raft_consensus.cc:729] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ea7ad8fe64bdcb4a852e9395ea150, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.564688 9572 consensus_queue.cc:260] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.564770 9572 ts_tablet_manager.cc:1428] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.564873 9578 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.564965 9578 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:36.685285 9578 raft_consensus.cc:482] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:36.685400 9578 raft_consensus.cc:504] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.685837 9578 leader_election.cc:288] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65:36563), 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741)
I20260923 22:17:36.688901 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.686108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:51571 (local address 127.1.129.65:36563)
0923 22:17:36.686161 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.686164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.686235 (+ 71us) server_negotiation.cc:407] Connection header received
0923 22:17:36.686334 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.686335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.686346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.686398 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:36.686604 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.687216 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.688181 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.688546 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.688600 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.688642 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.688782 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.688833 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:17:36.688907 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.686042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:36563 (local address 127.1.129.67:51571)
0923 22:17:36.686170 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.686180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.686265 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.686420 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.686420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.686422 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:36.686503 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.686505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.687307 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.687308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.688084 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.688085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.688633 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.688634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.688703 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.688707 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.688754 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.688787 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":14,"thread_start_us":69,"threads_started":1}
I20260923 22:17:36.688968 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.686117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:56941 (local address 127.1.129.66:33741)
0923 22:17:36.686161 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.686164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.686273 (+ 109us) server_negotiation.cc:407] Connection header received
0923 22:17:36.686314 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.686315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.686324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.686409 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:36.686568 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.687136 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.688181 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.688545 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.688651 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.688659 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.688842 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.688886 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:36.688982 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.686049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:33741 (local address 127.1.129.67:56941)
0923 22:17:36.686183 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.686192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.686265 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.686415 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.686416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.686418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:36.686496 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.686498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.687243 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.687244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.688083 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.688085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.688666 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.688667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.688760 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.688764 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.688821 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.688860 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":25,"thread_start_us":57,"threads_started":1}
I20260923 22:17:36.689188 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a909ea30a4c87937278d5722bf36e" candidate_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574" is_pre_election: true
I20260923 22:17:36.689298 9357 raft_consensus.cc:2454] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66ea7ad8fe64bdcb4a852e9395ea150 in term 0.
I20260923 22:17:36.689347 9217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a909ea30a4c87937278d5722bf36e" candidate_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5820834132c4546a6efb1374e6f3bc9" is_pre_election: true
I20260923 22:17:36.689512 9217 raft_consensus.cc:2454] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66ea7ad8fe64bdcb4a852e9395ea150 in term 0.
I20260923 22:17:36.689500 9430 leader_election.cc:302] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1396b309221441f2b4189cd198b16574, a66ea7ad8fe64bdcb4a852e9395ea150; no voters:
I20260923 22:17:36.689615 9578 raft_consensus.cc:2781] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:36.689661 9578 raft_consensus.cc:482] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:36.689682 9578 raft_consensus.cc:3037] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.690353 9578 raft_consensus.cc:504] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.690518 9578 leader_election.cc:288] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [CANDIDATE]: Term 1 election: Requested vote from peers c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65:36563), 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741)
I20260923 22:17:36.690660 9217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a909ea30a4c87937278d5722bf36e" candidate_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5820834132c4546a6efb1374e6f3bc9"
I20260923 22:17:36.690774 9217 raft_consensus.cc:3037] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.690762 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a909ea30a4c87937278d5722bf36e" candidate_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574"
I20260923 22:17:36.690834 9357 raft_consensus.cc:3037] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.691582 9217 raft_consensus.cc:2454] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a66ea7ad8fe64bdcb4a852e9395ea150 in term 1.
I20260923 22:17:36.691582 9357 raft_consensus.cc:2454] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a66ea7ad8fe64bdcb4a852e9395ea150 in term 1.
I20260923 22:17:36.691798 9430 leader_election.cc:302] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1396b309221441f2b4189cd198b16574, a66ea7ad8fe64bdcb4a852e9395ea150; no voters:
I20260923 22:17:36.691908 9578 raft_consensus.cc:2781] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:36.692054 9578 raft_consensus.cc:686] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 LEADER]: Becoming Leader. State: Replica: a66ea7ad8fe64bdcb4a852e9395ea150, State: Running, Role: LEADER
I20260923 22:17:36.692124 9578 consensus_queue.cc:237] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.692447 9613 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: TxnStatusTablet state changed. Reason: New leader a66ea7ad8fe64bdcb4a852e9395ea150. Latest consensus state: current_term: 1 leader_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.692620 9613 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:36.692811 9615 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 22:17:36.692822 9059 catalog_manager.cc:5495] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 reported cstate change: term changed from 0 to 1, leader changed from <none> to a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67). New cstate: current_term: 1 leader_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } health_report { overall_health: HEALTHY } } }
I20260923 22:17:36.692924 9615 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 22:17:36.708891 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.706211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.1.129.67:33745)
0923 22:17:36.706272 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.706273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.706281 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:36.706359 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.706359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.706376 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.706430 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:36.706550 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.707076 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.708174 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.708343 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.708391 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.708593 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:36.708680 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:36.708728 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.708811 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.708827 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:36.708940 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.706147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:33745 (local address 127.0.0.1:48404)
0923 22:17:36.706228 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.706240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.706281 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.706440 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.706441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.706442 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:36.706493 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.706495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.707154 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.707155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.708094 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.708096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.708416 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.708417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.708487 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.708503 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:36.708745 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:36.708753 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.708810 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.708858 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 22:17:36.710101 9357 raft_consensus.cc:1261] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Refusing update from remote peer a66ea7ad8fe64bdcb4a852e9395ea150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:36.710191 9217 raft_consensus.cc:1261] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Refusing update from remote peer a66ea7ad8fe64bdcb4a852e9395ea150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:36.710358 9613 consensus_queue.cc:1035] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:36.710497 9578 consensus_queue.cc:1035] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:36.711146 9579 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: TxnStatusTablet state changed. Reason: New leader a66ea7ad8fe64bdcb4a852e9395ea150. Latest consensus state: current_term: 1 leader_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.711230 9579 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:36.711176 9590 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: TxnStatusTablet state changed. Reason: New leader a66ea7ad8fe64bdcb4a852e9395ea150. Latest consensus state: current_term: 1 leader_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.711256 9590 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:36.712014 9578 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.712096 9578 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:36.712416 9578 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.712484 9578 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P a66ea7ad8fe64bdcb4a852e9395ea150: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:36.712524 9579 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.712607 9579 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P 1396b309221441f2b4189cd198b16574: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:36.712832 9585 mvcc.cc:205] Tried to move back new op lower bound from 7332666805082562560 to 7332666805011558400. Current Snapshot: MvccSnapshot[applied={T|T < 7332666805082562560}]
I20260923 22:17:36.712901 9590 tablet_replica.cc:439] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } } }
I20260923 22:17:36.712970 9590 tablet_replica.cc:442] T c41a909ea30a4c87937278d5722bf36e P c5820834132c4546a6efb1374e6f3bc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:36.718691 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.716149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:37251 (local address 127.1.129.67:33745)
0923 22:17:36.716211 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.716213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.716350 (+ 137us) server_negotiation.cc:407] Connection header received
0923 22:17:36.716378 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.716379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.716388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.716454 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:36.716540 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.717262 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.717963 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.718259 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.718327 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.718461 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:36.718494 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:36.718544 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.718590 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.718606 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:36.718765 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.716090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:33745 (local address 127.1.129.65:37251)
0923 22:17:36.716281 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.716290 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.716339 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.716455 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.716456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.716457 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:36.716496 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.716497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.717330 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.717331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.717883 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.717885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.718333 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.718334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.718374 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.718385 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:36.718552 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:36.718553 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.718587 (+ 34us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.718621 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260923 22:17:36.720544 9357 raft_consensus.cc:1261] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Refusing update from remote peer c5820834132c4546a6efb1374e6f3bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:36.720588 9497 raft_consensus.cc:1261] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Refusing update from remote peer c5820834132c4546a6efb1374e6f3bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:36.720743 9590 consensus_queue.cc:1035] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:36.720827 9582 consensus_queue.cc:1035] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:36.728636 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.726061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:59861 (local address 127.1.129.65:36563)
0923 22:17:36.726100 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.726102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.726248 (+ 146us) server_negotiation.cc:407] Connection header received
0923 22:17:36.726287 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.726288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.726296 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.726334 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:36.726464 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.726964 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.728079 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.728250 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.728301 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.728475 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:36.728497 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:36.728528 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.728578 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.728593 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:36.728878 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.726024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:36563 (local address 127.1.129.67:59861)
0923 22:17:36.726168 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.726179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.726239 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.726345 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.726346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.726347 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:36.726397 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.726399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.727035 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.727036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.727999 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.728001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.728322 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.728323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.728387 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.728401 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:36.728530 (+ 129us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:36.728531 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.728578 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.728612 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260923 22:17:36.737414 9059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:36.737519 9059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:36.738142 9059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: Sending DeleteTablet for 3 replicas of tablet 49232b49b8944477b494e64e513dc90c
I20260923 22:17:36.738374 9337 tablet_service.cc:1514] Processing DeleteTablet for tablet 49232b49b8944477b494e64e513dc90c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:43918
I20260923 22:17:36.738484 9596 tablet_replica.cc:330] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: stopping tablet replica
I20260923 22:17:36.738514 9197 tablet_service.cc:1514] Processing DeleteTablet for tablet 49232b49b8944477b494e64e513dc90c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:41592
I20260923 22:17:36.738547 9596 raft_consensus.cc:2229] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:36.738586 9596 raft_consensus.cc:2258] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.738598 9597 tablet_replica.cc:330] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: stopping tablet replica
I20260923 22:17:36.738667 9597 raft_consensus.cc:2229] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:36.738722 9597 raft_consensus.cc:2258] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.738726 9477 tablet_service.cc:1514] Processing DeleteTablet for tablet 49232b49b8944477b494e64e513dc90c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:48400
I20260923 22:17:36.738960 9596 ts_tablet_manager.cc:1905] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.739032 9597 ts_tablet_manager.cc:1905] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.740438 9596 ts_tablet_manager.cc:1918] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:36.740507 9596 log.cc:1196] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/49232b49b8944477b494e64e513dc90c
I20260923 22:17:36.740513 9595 tablet_replica.cc:330] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: stopping tablet replica
I20260923 22:17:36.740574 9595 raft_consensus.cc:2229] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:36.740629 9595 raft_consensus.cc:2258] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.740726 9596 ts_tablet_manager.cc:1939] T 49232b49b8944477b494e64e513dc90c P 1396b309221441f2b4189cd198b16574: Deleting consensus metadata
I20260923 22:17:36.740833 9597 ts_tablet_manager.cc:1918] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:36.740885 9597 log.cc:1196] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/49232b49b8944477b494e64e513dc90c
I20260923 22:17:36.741097 9597 ts_tablet_manager.cc:1939] T 49232b49b8944477b494e64e513dc90c P c5820834132c4546a6efb1374e6f3bc9: Deleting consensus metadata
I20260923 22:17:36.741199 9046 catalog_manager.cc:4841] TS 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741): tablet 49232b49b8944477b494e64e513dc90c (table test-table [id=798614332d2048a596cd4de90875a854]) successfully deleted
I20260923 22:17:36.741266 9595 ts_tablet_manager.cc:1905] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.741511 9047 catalog_manager.cc:4841] TS c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65:36563): tablet 49232b49b8944477b494e64e513dc90c (table test-table [id=798614332d2048a596cd4de90875a854]) successfully deleted
I20260923 22:17:36.742692 9595 ts_tablet_manager.cc:1918] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:36.742754 9595 log.cc:1196] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/49232b49b8944477b494e64e513dc90c
I20260923 22:17:36.742985 9595 ts_tablet_manager.cc:1939] T 49232b49b8944477b494e64e513dc90c P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting consensus metadata
I20260923 22:17:36.743290 9047 catalog_manager.cc:4841] TS a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745): tablet 49232b49b8944477b494e64e513dc90c (table test-table [id=798614332d2048a596cd4de90875a854]) successfully deleted
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 22:17:36.762614 9109 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:36.756325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.1.129.126:41621)
0923 22:17:36.756401 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.756404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.756533 (+ 129us) server_negotiation.cc:407] Connection header received
0923 22:17:36.756605 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.756606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.756615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.756664 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.757351 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.757822 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.758963 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.759145 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.759198 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.762546 (+ 3348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47342: BlockingRecv error: recv got EOF from 127.0.0.1:47342 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 22:17:36 dist-test-slave-dmw7 krb5kdc[9015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM
I20260923 22:17:36.770742 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.763459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.1.129.126:41621)
0923 22:17:36.763496 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.763498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.763560 (+ 62us) server_negotiation.cc:407] Connection header received
0923 22:17:36.763608 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.763609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.763618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.763678 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.764102 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.764860 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.765555 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.765840 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.765914 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.768084 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.768086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.768086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.768094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.769625 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.769967 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.769968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.769968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.770031 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.770246 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.770247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.770248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.770382 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.770481 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.770562 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.770651 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:36.772265 9059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:36.772437 9059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:36.773763 9477 tablet_service.cc:1467] Processing CreateTablet for tablet f77a60e45d294647b4e57a46235063d7 (DEFAULT_TABLE table=test-table [id=ae7586255d9e4734a2ffdec7ea026bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.773808 9197 tablet_service.cc:1467] Processing CreateTablet for tablet f77a60e45d294647b4e57a46235063d7 (DEFAULT_TABLE table=test-table [id=ae7586255d9e4734a2ffdec7ea026bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.773919 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77a60e45d294647b4e57a46235063d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.773934 9197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77a60e45d294647b4e57a46235063d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.774809 9337 tablet_service.cc:1467] Processing CreateTablet for tablet f77a60e45d294647b4e57a46235063d7 (DEFAULT_TABLE table=test-table [id=ae7586255d9e4734a2ffdec7ea026bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.774932 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77a60e45d294647b4e57a46235063d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.775007 9572 tablet_bootstrap.cc:492] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: Bootstrap starting.
I20260923 22:17:36.775277 9572 tablet_bootstrap.cc:654] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.775377 9571 tablet_bootstrap.cc:492] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: Bootstrap starting.
I20260923 22:17:36.775686 9571 tablet_bootstrap.cc:654] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.775921 9572 tablet_bootstrap.cc:492] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: No bootstrap required, opened a new log
I20260923 22:17:36.775967 9572 ts_tablet_manager.cc:1397] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:36.776067 9572 raft_consensus.cc:348] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.776144 9572 raft_consensus.cc:374] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.776163 9572 raft_consensus.cc:729] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ea7ad8fe64bdcb4a852e9395ea150, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.776206 9572 consensus_queue.cc:260] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.776295 9572 ts_tablet_manager.cc:1428] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.776358 9573 tablet_bootstrap.cc:492] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: Bootstrap starting.
I20260923 22:17:36.776762 9573 tablet_bootstrap.cc:654] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.777450 9573 tablet_bootstrap.cc:492] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: No bootstrap required, opened a new log
I20260923 22:17:36.777508 9573 ts_tablet_manager.cc:1397] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.777650 9573 raft_consensus.cc:348] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.777724 9573 raft_consensus.cc:374] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.777746 9573 raft_consensus.cc:729] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1396b309221441f2b4189cd198b16574, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.777793 9573 consensus_queue.cc:260] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.777815 9571 tablet_bootstrap.cc:492] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: No bootstrap required, opened a new log
I20260923 22:17:36.777858 9571 ts_tablet_manager.cc:1397] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:36.777879 9573 ts_tablet_manager.cc:1428] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:36.777947 9571 raft_consensus.cc:348] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.777992 9571 raft_consensus.cc:374] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.778005 9571 raft_consensus.cc:729] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.778069 9571 consensus_queue.cc:260] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.778134 9590 raft_consensus.cc:482] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:36.778223 9590 raft_consensus.cc:504] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.778368 9590 leader_election.cc:288] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741), a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745)
I20260923 22:17:36.778468 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77a60e45d294647b4e57a46235063d7" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574" is_pre_election: true
I20260923 22:17:36.778537 9357 raft_consensus.cc:2454] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 0.
I20260923 22:17:36.778610 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77a60e45d294647b4e57a46235063d7" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" is_pre_election: true
I20260923 22:17:36.778695 9497 raft_consensus.cc:2454] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 0.
I20260923 22:17:36.778750 9571 ts_tablet_manager.cc:1428] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.778831 9151 leader_election.cc:302] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a66ea7ad8fe64bdcb4a852e9395ea150, c5820834132c4546a6efb1374e6f3bc9; no voters:
I20260923 22:17:36.778932 9582 raft_consensus.cc:2781] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:36.778960 9582 raft_consensus.cc:482] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:36.778985 9582 raft_consensus.cc:3037] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.779548 9582 raft_consensus.cc:504] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.779685 9582 leader_election.cc:288] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741), a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745)
I20260923 22:17:36.779837 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77a60e45d294647b4e57a46235063d7" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574"
I20260923 22:17:36.779915 9357 raft_consensus.cc:3037] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.780380 9357 raft_consensus.cc:2454] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 1.
I20260923 22:17:36.780542 9150 leader_election.cc:302] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1396b309221441f2b4189cd198b16574, c5820834132c4546a6efb1374e6f3bc9; no voters:
I20260923 22:17:36.780644 9590 raft_consensus.cc:2781] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:36.780660 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77a60e45d294647b4e57a46235063d7" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150"
I20260923 22:17:36.780704 9590 raft_consensus.cc:686] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 1 LEADER]: Becoming Leader. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Running, Role: LEADER
I20260923 22:17:36.780723 9497 raft_consensus.cc:3037] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.780745 9590 consensus_queue.cc:237] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.781198 9059 catalog_manager.cc:5495] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "c5820834132c4546a6efb1374e6f3bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:36.781507 9497 raft_consensus.cc:2454] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 1.
I20260923 22:17:36.788293 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.785373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.1.129.65:36563)
0923 22:17:36.785418 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.785420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.785455 (+ 35us) server_negotiation.cc:407] Connection header received
0923 22:17:36.785519 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.785520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.785532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.785601 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:36.785689 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.786413 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.787439 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.787715 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.787796 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.787942 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:36.787978 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:36.788038 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.788189 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.788216 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:36.789132 9497 raft_consensus.cc:1261] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Refusing update from remote peer c5820834132c4546a6efb1374e6f3bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:36.789202 9357 raft_consensus.cc:1261] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Refusing update from remote peer c5820834132c4546a6efb1374e6f3bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:36.789354 9577 consensus_queue.cc:1035] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:36.789579 9577 consensus_queue.cc:1035] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:36.793046 9060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:36.795837 9060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:36.796495 9060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: Sending DeleteTablet for 3 replicas of tablet f77a60e45d294647b4e57a46235063d7
I20260923 22:17:36.796805 9477 tablet_service.cc:1514] Processing DeleteTablet for tablet f77a60e45d294647b4e57a46235063d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:48400
I20260923 22:17:36.796916 9595 tablet_replica.cc:330] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: stopping tablet replica
I20260923 22:17:36.796986 9060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:36.797116 9060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:36.797156 9197 tablet_service.cc:1514] Processing DeleteTablet for tablet f77a60e45d294647b4e57a46235063d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:41592
I20260923 22:17:36.797224 9597 tablet_replica.cc:330] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: stopping tablet replica
I20260923 22:17:36.797267 9597 raft_consensus.cc:2229] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:36.797329 9597 raft_consensus.cc:2258] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.796967 9595 raft_consensus.cc:2229] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:36.797438 9595 raft_consensus.cc:2258] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.797963 9337 tablet_service.cc:1514] Processing DeleteTablet for tablet f77a60e45d294647b4e57a46235063d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:36 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:43918
I20260923 22:17:36.798053 9596 tablet_replica.cc:330] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: stopping tablet replica
I20260923 22:17:36.798096 9596 raft_consensus.cc:2229] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:36.798149 9596 raft_consensus.cc:2258] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.798403 9197 tablet_service.cc:1467] Processing CreateTablet for tablet f9d3cfc7942541df884b60702190e6ed (DEFAULT_TABLE table=test-table [id=98c086cc7f6540b5873933eaf9f8fdc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.798455 9597 ts_tablet_manager.cc:1905] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.798535 9197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d3cfc7942541df884b60702190e6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.798708 9477 tablet_service.cc:1467] Processing CreateTablet for tablet f9d3cfc7942541df884b60702190e6ed (DEFAULT_TABLE table=test-table [id=98c086cc7f6540b5873933eaf9f8fdc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.798805 9595 ts_tablet_manager.cc:1905] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.798818 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d3cfc7942541df884b60702190e6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.798992 9337 tablet_service.cc:1467] Processing CreateTablet for tablet f9d3cfc7942541df884b60702190e6ed (DEFAULT_TABLE table=test-table [id=98c086cc7f6540b5873933eaf9f8fdc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:36.799109 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d3cfc7942541df884b60702190e6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.800100 9572 tablet_bootstrap.cc:492] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: Bootstrap starting.
I20260923 22:17:36.800462 9573 tablet_bootstrap.cc:492] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: Bootstrap starting.
I20260923 22:17:36.800530 9572 tablet_bootstrap.cc:654] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.801077 9573 tablet_bootstrap.cc:654] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.802680 9595 ts_tablet_manager.cc:1918] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:36.802733 9595 log.cc:1196] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/f77a60e45d294647b4e57a46235063d7
I20260923 22:17:36.802814 9571 tablet_bootstrap.cc:492] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9: Bootstrap starting.
I20260923 22:17:36.802949 9595 ts_tablet_manager.cc:1939] T f77a60e45d294647b4e57a46235063d7 P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting consensus metadata
I20260923 22:17:36.803090 9571 tablet_bootstrap.cc:654] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.803304 9597 ts_tablet_manager.cc:1918] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:36.803355 9597 log.cc:1196] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/f77a60e45d294647b4e57a46235063d7
I20260923 22:17:36.803382 9047 catalog_manager.cc:4841] TS a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745): tablet f77a60e45d294647b4e57a46235063d7 (table test-table [id=ae7586255d9e4734a2ffdec7ea026bbb]) successfully deleted
I20260923 22:17:36.803439 9572 tablet_bootstrap.cc:492] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: No bootstrap required, opened a new log
I20260923 22:17:36.803478 9572 ts_tablet_manager.cc:1397] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 22:17:36.803562 9597 ts_tablet_manager.cc:1939] T f77a60e45d294647b4e57a46235063d7 P c5820834132c4546a6efb1374e6f3bc9: Deleting consensus metadata
I20260923 22:17:36.803613 9572 raft_consensus.cc:348] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.803650 9596 ts_tablet_manager.cc:1905] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.803682 9572 raft_consensus.cc:374] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.803709 9572 raft_consensus.cc:729] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ea7ad8fe64bdcb4a852e9395ea150, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.803753 9572 consensus_queue.cc:260] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.803840 9572 ts_tablet_manager.cc:1428] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.805006 9596 ts_tablet_manager.cc:1918] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:36.805068 9596 log.cc:1196] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/f77a60e45d294647b4e57a46235063d7
I20260923 22:17:36.805294 9596 ts_tablet_manager.cc:1939] T f77a60e45d294647b4e57a46235063d7 P 1396b309221441f2b4189cd198b16574: Deleting consensus metadata
I20260923 22:17:36.805539 9047 catalog_manager.cc:4841] TS c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65:36563): tablet f77a60e45d294647b4e57a46235063d7 (table test-table [id=ae7586255d9e4734a2ffdec7ea026bbb]) successfully deleted
I20260923 22:17:36.805720 9046 catalog_manager.cc:4841] TS 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741): tablet f77a60e45d294647b4e57a46235063d7 (table test-table [id=ae7586255d9e4734a2ffdec7ea026bbb]) successfully deleted
I20260923 22:17:36.805994 9571 tablet_bootstrap.cc:492] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9: No bootstrap required, opened a new log
I20260923 22:17:36.806038 9573 tablet_bootstrap.cc:492] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: No bootstrap required, opened a new log
I20260923 22:17:36.806059 9571 ts_tablet_manager.cc:1397] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260923 22:17:36.806080 9573 ts_tablet_manager.cc:1397] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260923 22:17:36.806170 9571 raft_consensus.cc:348] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.806237 9571 raft_consensus.cc:374] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.806267 9571 raft_consensus.cc:729] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.806227 9573 raft_consensus.cc:348] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.806290 9573 raft_consensus.cc:374] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.806320 9573 raft_consensus.cc:729] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1396b309221441f2b4189cd198b16574, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.806306 9571 consensus_queue.cc:260] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.806363 9573 consensus_queue.cc:260] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:36.806384 9571 ts_tablet_manager.cc:1428] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:36.806442 9573 ts_tablet_manager.cc:1428] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:37.164597 9577 raft_consensus.cc:482] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:37.164690 9577 raft_consensus.cc:504] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:37.164916 9577 leader_election.cc:288] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741), a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745)
I20260923 22:17:37.165158 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d3cfc7942541df884b60702190e6ed" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" is_pre_election: true
I20260923 22:17:37.165158 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d3cfc7942541df884b60702190e6ed" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574" is_pre_election: true
I20260923 22:17:37.165258 9497 raft_consensus.cc:2454] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 0.
I20260923 22:17:37.165269 9357 raft_consensus.cc:2454] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 0.
I20260923 22:17:37.165441 9151 leader_election.cc:302] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a66ea7ad8fe64bdcb4a852e9395ea150, c5820834132c4546a6efb1374e6f3bc9; no voters:
I20260923 22:17:37.165565 9577 raft_consensus.cc:2781] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:37.165609 9577 raft_consensus.cc:482] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:37.165632 9577 raft_consensus.cc:3037] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:37.166246 9577 raft_consensus.cc:504] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:37.166384 9577 leader_election.cc:288] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741), a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745)
I20260923 22:17:37.166502 9357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d3cfc7942541df884b60702190e6ed" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1396b309221441f2b4189cd198b16574"
I20260923 22:17:37.166551 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d3cfc7942541df884b60702190e6ed" candidate_uuid: "c5820834132c4546a6efb1374e6f3bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150"
I20260923 22:17:37.166603 9357 raft_consensus.cc:3037] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:37.166666 9497 raft_consensus.cc:3037] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:37.167222 9497 raft_consensus.cc:2454] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 1.
I20260923 22:17:37.167323 9357 raft_consensus.cc:2454] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5820834132c4546a6efb1374e6f3bc9 in term 1.
I20260923 22:17:37.167425 9151 leader_election.cc:302] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a66ea7ad8fe64bdcb4a852e9395ea150, c5820834132c4546a6efb1374e6f3bc9; no voters:
I20260923 22:17:37.167510 9577 raft_consensus.cc:2781] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:37.167560 9577 raft_consensus.cc:686] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 1 LEADER]: Becoming Leader. State: Replica: c5820834132c4546a6efb1374e6f3bc9, State: Running, Role: LEADER
I20260923 22:17:37.167615 9577 consensus_queue.cc:237] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } }
I20260923 22:17:37.168190 9058 catalog_manager.cc:5495] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "c5820834132c4546a6efb1374e6f3bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820834132c4546a6efb1374e6f3bc9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:37.247155 9497 raft_consensus.cc:1261] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Refusing update from remote peer c5820834132c4546a6efb1374e6f3bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:37.247157 9357 raft_consensus.cc:1261] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Refusing update from remote peer c5820834132c4546a6efb1374e6f3bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:37.247404 9582 consensus_queue.cc:1035] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66ea7ad8fe64bdcb4a852e9395ea150" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:37.247537 9577 consensus_queue.cc:1035] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1396b309221441f2b4189cd198b16574" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:37.256779 9059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:37.256878 9059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:37.257442 9059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88658f5a52c740308540176a0f2d7274: Sending DeleteTablet for 3 replicas of tablet f9d3cfc7942541df884b60702190e6ed
I20260923 22:17:37.257715 9337 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d3cfc7942541df884b60702190e6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:37 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:43918
I20260923 22:17:37.257838 9197 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d3cfc7942541df884b60702190e6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:37 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:41592
I20260923 22:17:37.257925 9596 tablet_replica.cc:330] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: stopping tablet replica
I20260923 22:17:37.257990 9596 raft_consensus.cc:2229] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:37.258029 9596 raft_consensus.cc:2258] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:37.258016 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 9132
I20260923 22:17:37.258092 9597 tablet_replica.cc:330] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9: stopping tablet replica
I20260923 22:17:37.258140 9597 raft_consensus.cc:2229] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:37.258190 9597 raft_consensus.cc:2258] T f9d3cfc7942541df884b60702190e6ed P c5820834132c4546a6efb1374e6f3bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:37.259629 9477 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d3cfc7942541df884b60702190e6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:37 UTC) from {username='oryx', principal='oryx/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:48400
I20260923 22:17:37.259740 9595 tablet_replica.cc:330] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: stopping tablet replica
I20260923 22:17:37.259796 9595 raft_consensus.cc:2229] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:37.259835 9595 raft_consensus.cc:2258] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:37.259982 9596 ts_tablet_manager.cc:1905] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:37.260151 9595 ts_tablet_manager.cc:1905] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:37.261830 9595 ts_tablet_manager.cc:1918] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:37.261849 9596 ts_tablet_manager.cc:1918] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:37.261893 9595 log.cc:1196] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/f9d3cfc7942541df884b60702190e6ed
I20260923 22:17:37.261904 9596 log.cc:1196] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/f9d3cfc7942541df884b60702190e6ed
I20260923 22:17:37.262121 9596 ts_tablet_manager.cc:1939] T f9d3cfc7942541df884b60702190e6ed P 1396b309221441f2b4189cd198b16574: Deleting consensus metadata
I20260923 22:17:37.262121 9595 ts_tablet_manager.cc:1939] T f9d3cfc7942541df884b60702190e6ed P a66ea7ad8fe64bdcb4a852e9395ea150: Deleting consensus metadata
I20260923 22:17:37.262485 9046 catalog_manager.cc:4841] TS 1396b309221441f2b4189cd198b16574 (127.1.129.66:33741): tablet f9d3cfc7942541df884b60702190e6ed (table test-table [id=98c086cc7f6540b5873933eaf9f8fdc8]) successfully deleted
W20260923 22:17:37.265192 9047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:36563 (error 108)
I20260923 22:17:37.265442 9047 catalog_manager.cc:4841] TS a66ea7ad8fe64bdcb4a852e9395ea150 (127.1.129.67:33745): tablet f9d3cfc7942541df884b60702190e6ed (table test-table [id=98c086cc7f6540b5873933eaf9f8fdc8]) successfully deleted
I20260923 22:17:37.265867 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 9272
I20260923 22:17:37.265895 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.265601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:36563 (local address 127.0.0.1:41618)
0923 22:17:37.265806 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.265853 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:36563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":38,"threads_started":1}
W20260923 22:17:37.266541 9047 catalog_manager.cc:4584] TS c5820834132c4546a6efb1374e6f3bc9 (127.1.129.65:36563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9d3cfc7942541df884b60702190e6ed: Network error: Client connection negotiation failed: client connection to 127.1.129.65:36563: connect: Connection refused (error 111)
I20260923 22:17:37.271097 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 9412
I20260923 22:17:37.276458 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 9028
2026-09-23T22:17:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2984 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:37 dist-test-slave-dmw7 krb5kdc[9663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:37 dist-test-slave-dmw7 krb5kdc[9663](info): set up 2 sockets
Sep 23 22:17:37 dist-test-slave-dmw7 krb5kdc[9663](info): commencing operation
krb5kdc: starting...
W20260923 22:17:38.319262 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:38 dist-test-slave-dmw7 krb5kdc[9663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:38.438081 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36985
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:33825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:38.516916 9676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:38.517099 9676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:38.517120 9676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:38.518169 9676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:38.518203 9676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:38.518216 9676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:38.518229 9676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:38.518241 9676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:38.519646 9676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:33825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:38.519820 9676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:38.519972 9676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:38.524823 9676 server_base.cc:1034] running on GCE node
W20260923 22:17:38.524798 9681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:38.524778 9684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:38.524775 9682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:38.525177 9676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:38.525465 9676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:38.526710 9676 hybrid_clock.cc:648] HybridClock initialized: now 1790201858526685 us; error 45 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T22:17:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1257 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:38 dist-test-slave-dmw7 krb5kdc[9689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:38 dist-test-slave-dmw7 krb5kdc[9689](info): set up 2 sockets
Sep 23 22:17:38 dist-test-slave-dmw7 krb5kdc[9689](info): commencing operation
krb5kdc: starting...
W20260923 22:17:39.567175 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:39 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:39.686239 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:39913
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:39913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:39.769214 9702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:39.769472 9702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:39.769508 9702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:39.770695 9702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:39.770771 9702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:39.770797 9702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:39.770816 9702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:39.770846 9702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:39.772300 9702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:39913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:39.772583 9702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:39.772816 9702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:39.777750 9708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:39.777896 9702 server_base.cc:1034] running on GCE node
W20260923 22:17:39.777740 9707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:39.777743 9710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:39.778200 9702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:39.778520 9702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:39.779701 9702 hybrid_clock.cc:648] HybridClock initialized: now 1790201859779680 us; error 40 us; skew 500 ppm
Sep 23 22:17:39 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:39.782245 9702 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:39.782789 9702 webserver.cc:460] Webserver started at http://127.1.129.126:44881/ using document root <none> and password file <none>
I20260923 22:17:39.783025 9702 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:39.783092 9702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:39.783226 9702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:39.784277 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "17e88933e14a44afb138f091c62ad946"
format_stamp: "Formatted at 2026-09-23 22:17:39 on dist-test-slave-dmw7"
server_key: "ee7e9f39d772ddd6a83706000962f920"
server_key_iv: "ed6fb5b9176756cc21917381ffadde49"
server_key_version: "encryptionkey@0"
I20260923 22:17:39.784643 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "17e88933e14a44afb138f091c62ad946"
format_stamp: "Formatted at 2026-09-23 22:17:39 on dist-test-slave-dmw7"
server_key: "ee7e9f39d772ddd6a83706000962f920"
server_key_iv: "ed6fb5b9176756cc21917381ffadde49"
server_key_version: "encryptionkey@0"
I20260923 22:17:39.785945 9702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:39.786684 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:39.786844 9702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:39.786942 9702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "17e88933e14a44afb138f091c62ad946"
format_stamp: "Formatted at 2026-09-23 22:17:39 on dist-test-slave-dmw7"
server_key: "ee7e9f39d772ddd6a83706000962f920"
server_key_iv: "ed6fb5b9176756cc21917381ffadde49"
server_key_version: "encryptionkey@0"
I20260923 22:17:39.787025 9702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:39.795598 9702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:39.797343 9702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:39.797505 9702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:39.801734 9702 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:39913
I20260923 22:17:39.801764 9769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:39913 every 8 connection(s)
I20260923 22:17:39.802104 9702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:39.803022 9770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:39.805150 9770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Bootstrap starting.
I20260923 22:17:39.805671 9770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:39.805914 9770 log.cc:824] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:39.806567 9770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: No bootstrap required, opened a new log
I20260923 22:17:39.808184 9770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } }
I20260923 22:17:39.808322 9770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:39.808344 9770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e88933e14a44afb138f091c62ad946, State: Initialized, Role: FOLLOWER
I20260923 22:17:39.808492 9770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } }
I20260923 22:17:39.808573 9770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:39.808596 9770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:39.808624 9770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:39.809101 9770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } }
I20260923 22:17:39.809182 9770 leader_election.cc:302] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17e88933e14a44afb138f091c62ad946; no voters:
I20260923 22:17:39.809303 9770 leader_election.cc:288] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:39.809361 9773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:39.809602 9770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:39.809749 9773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 1 LEADER]: Becoming Leader. State: Replica: 17e88933e14a44afb138f091c62ad946, State: Running, Role: LEADER
I20260923 22:17:39.809844 9773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } }
I20260923 22:17:39.810259 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17e88933e14a44afb138f091c62ad946. Latest consensus state: current_term: 1 leader_uuid: "17e88933e14a44afb138f091c62ad946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } } }
I20260923 22:17:39.810297 9774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17e88933e14a44afb138f091c62ad946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } } }
I20260923 22:17:39.810465 9774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:39.810842 9782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:39.810990 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 9702
I20260923 22:17:39.811111 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:39.811301 1541 external_mini_cluster.cc:1428] Setting key c454b513fd58f7fc821d2c2a2348d30a
I20260923 22:17:39.811301 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:39.811405 9782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:39.812942 9782 catalog_manager.cc:1347] Generated new cluster ID: 90836c0f756d40e886c3a9ecd9fe286f
I20260923 22:17:39.812990 9782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:39 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:39.823194 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.811782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.1.129.126:39913)
0923 22:17:39.811963 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:39.811972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:39.812004 (+ 32us) server_negotiation.cc:407] Connection header received
0923 22:17:39.812450 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:39.812456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:39.812515 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:39.812648 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:39.813509 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.814910 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.815990 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.816599 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.816698 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:39.819576 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:39.819580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:39.819580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:39.819593 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:39.821530 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:39.821915 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:39.821918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:39.821918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:39.822029 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:39.822229 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:39.822230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:39.822230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:39.822551 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:39.822719 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:39.822813 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 22:17:39.822961 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260923 22:17:39.851305 9782 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:39.852079 9782 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:39.863981 9782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Generated new TSK 0
I20260923 22:17:39.864194 9782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:39.893780 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:39913
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:39.973659 9796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:39.973865 9796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:39.973887 9796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:39.975095 9796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:39.975135 9796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:39.975160 9796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:39.976553 9796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:39.976801 9796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:39.977025 9796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:39.982092 9804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:39.982136 9796 server_base.cc:1034] running on GCE node
W20260923 22:17:39.982228 9802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:39.982301 9801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:39.982546 9796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:39.982836 9796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:39.983987 9796 hybrid_clock.cc:648] HybridClock initialized: now 1790201859983976 us; error 28 us; skew 500 ppm
Sep 23 22:17:39 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:39.991595 9796 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:39.992071 9796 webserver.cc:460] Webserver started at http://127.1.129.65:36981/ using document root <none> and password file <none>
I20260923 22:17:39.992311 9796 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:39.992386 9796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:39.992542 9796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:39.993603 9796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "789ae423770048a49f1cd090161ccfc6"
format_stamp: "Formatted at 2026-09-23 22:17:39 on dist-test-slave-dmw7"
server_key: "f4ca3e947f3e867ed4bb297a272b4e11"
server_key_iv: "ce2fb608d5f5d4eb31ba787a3898e7f2"
server_key_version: "encryptionkey@0"
I20260923 22:17:39.993980 9796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "789ae423770048a49f1cd090161ccfc6"
format_stamp: "Formatted at 2026-09-23 22:17:39 on dist-test-slave-dmw7"
server_key: "f4ca3e947f3e867ed4bb297a272b4e11"
server_key_iv: "ce2fb608d5f5d4eb31ba787a3898e7f2"
server_key_version: "encryptionkey@0"
I20260923 22:17:39.995414 9796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:39.996183 9811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:39.996358 9796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:39.996459 9796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "789ae423770048a49f1cd090161ccfc6"
format_stamp: "Formatted at 2026-09-23 22:17:39 on dist-test-slave-dmw7"
server_key: "f4ca3e947f3e867ed4bb297a272b4e11"
server_key_iv: "ce2fb608d5f5d4eb31ba787a3898e7f2"
server_key_version: "encryptionkey@0"
I20260923 22:17:39.996542 9796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.011333 9796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.013550 9796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.013752 9796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.014021 9796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:40.014369 9796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:40.014436 9796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.014472 9796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:40.014500 9796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.020574 9796 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:33677
I20260923 22:17:40.020622 9924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:33677 every 8 connection(s)
I20260923 22:17:40.020939 9796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:40.028671 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 9796
I20260923 22:17:40.028795 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:40.028988 1541 external_mini_cluster.cc:1428] Setting key dee014be5514ac54fe9103500d01643b
I20260923 22:17:40.030371 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.021704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:37471 (local address 127.1.129.126:39913)
0923 22:17:40.021797 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.021799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.021970 (+ 171us) server_negotiation.cc:407] Connection header received
0923 22:17:40.022482 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.022483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.022494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.022546 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.023515 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.024005 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.024972 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.025166 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.025216 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.027340 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.027341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.027342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.027349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.028828 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.029328 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.029329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.029330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.029393 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.029685 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.029686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.029686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.029806 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.029887 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.030192 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.030279 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 22:17:40.030571 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.021650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:37471)
0923 22:17:40.021843 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.021879 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.022370 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.022605 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.022609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.023240 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.023430 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.023435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.024092 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.024093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.024899 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.024902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.025264 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.025265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.025377 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.025844 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.025852 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.027248 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:40.028945 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.028950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.028952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.029254 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.029479 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.029480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.029480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.029613 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.029920 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:40.029926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:40.030052 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 22:17:40.030169 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 22:17:40.030336 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":112,"threads_started":1}
I20260923 22:17:40.031162 9925 heartbeater.cc:347] Connected to a master server at 127.1.129.126:39913
I20260923 22:17:40.031265 9925 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:40.031455 9925 heartbeater.cc:511] Master 127.1.129.126:39913 requested a full tablet report, sending...
I20260923 22:17:40.031930 9734 ts_manager.cc:194] Registered new tserver with Master: 789ae423770048a49f1cd090161ccfc6 (127.1.129.65:33677)
I20260923 22:17:40.032859 9734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:37471
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:40.053610 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:39913
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:40.131888 9930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.132082 9930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.132107 9930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.133471 9930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.133548 9930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.133589 9930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:40.135161 9930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:40.135357 9930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.135525 9930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:40.140455 9938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.140461 9935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.140671 9930 server_base.cc:1034] running on GCE node
W20260923 22:17:40.140666 9936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.140985 9930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.141215 9930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.142387 9930 hybrid_clock.cc:648] HybridClock initialized: now 1790201860142373 us; error 40 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.144824 9930 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:40.145298 9930 webserver.cc:460] Webserver started at http://127.1.129.66:40045/ using document root <none> and password file <none>
I20260923 22:17:40.145540 9930 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.145612 9930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.145756 9930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:40.146842 9930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "d870419a8da4499ab0d35428dcb6d16b"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-dmw7"
server_key: "cc17df3513b21fa3b4586507cee509d3"
server_key_iv: "609087953a326d7df9a65254ddc9b9d0"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.147192 9930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "d870419a8da4499ab0d35428dcb6d16b"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-dmw7"
server_key: "cc17df3513b21fa3b4586507cee509d3"
server_key_iv: "609087953a326d7df9a65254ddc9b9d0"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.148487 9930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:40.149247 9945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.149447 9930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:40.149545 9930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "d870419a8da4499ab0d35428dcb6d16b"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-dmw7"
server_key: "cc17df3513b21fa3b4586507cee509d3"
server_key_iv: "609087953a326d7df9a65254ddc9b9d0"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.149624 9930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.160595 9930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.162246 9930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.162410 9930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.162616 9930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:40.162943 9930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:40.162999 9930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.163034 9930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:40.163085 9930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.168879 9930 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:34479
I20260923 22:17:40.168917 10058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:34479 every 8 connection(s)
I20260923 22:17:40.169210 9930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:40.177971 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.169951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:43079 (local address 127.1.129.126:39913)
0923 22:17:40.170031 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.170034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.170221 (+ 187us) server_negotiation.cc:407] Connection header received
0923 22:17:40.170734 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.170735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.170746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.170801 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.171674 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.172161 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.173085 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.173278 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.173327 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.175391 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.175392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.175393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.175401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.176606 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.177026 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.177028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.177028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.177087 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.177337 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.177338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.177338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.177454 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.177533 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.177793 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.177878 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 22:17:40.178169 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.169927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:43079)
0923 22:17:40.170101 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.170135 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.170583 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.170869 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.170872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.171411 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.171596 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.171600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.172240 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.172240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.173009 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.173011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.173349 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.173350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.173448 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.173882 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.173890 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.175283 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:40.176697 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.176703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.176706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.176929 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.177158 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.177159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.177159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.177263 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.177523 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:40.177529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:40.177663 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 22:17:40.177783 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 22:17:40.177936 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260923 22:17:40.178243 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 9930
I20260923 22:17:40.178357 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:40.178550 1541 external_mini_cluster.cc:1428] Setting key e63df51f399835899e724f2de4cf23f9
I20260923 22:17:40.178705 10059 heartbeater.cc:347] Connected to a master server at 127.1.129.126:39913
I20260923 22:17:40.178807 10059 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:40.178980 10059 heartbeater.cc:511] Master 127.1.129.126:39913 requested a full tablet report, sending...
I20260923 22:17:40.179351 9734 ts_manager.cc:194] Registered new tserver with Master: d870419a8da4499ab0d35428dcb6d16b (127.1.129.66:34479)
I20260923 22:17:40.179854 9734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:43079
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:40.205113 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:39913
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:40.284411 10064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.284658 10064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.284696 10064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.285862 10064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.285945 10064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.285990 10064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:40.287439 10064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:40.287709 10064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.287936 10064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:40.293004 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.293058 10064 server_base.cc:1034] running on GCE node
W20260923 22:17:40.293107 10069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.293184 10070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.293497 10064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.293749 10064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.294922 10064 hybrid_clock.cc:648] HybridClock initialized: now 1790201860294902 us; error 38 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.297439 10064 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:40.297823 10064 webserver.cc:460] Webserver started at http://127.1.129.67:44437/ using document root <none> and password file <none>
I20260923 22:17:40.298018 10064 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.298058 10064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.298143 10064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:40.299212 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "3431aa9cdb53452c8dbc6cf23e0581e1"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-dmw7"
server_key: "fc25ba458b56fc45d54ea6b584bc89b6"
server_key_iv: "dddb6033e845d2057e177c8f327792f3"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.299666 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "3431aa9cdb53452c8dbc6cf23e0581e1"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-dmw7"
server_key: "fc25ba458b56fc45d54ea6b584bc89b6"
server_key_iv: "dddb6033e845d2057e177c8f327792f3"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.301038 10064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:40.301673 10079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.301853 10064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:40.301920 10064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "3431aa9cdb53452c8dbc6cf23e0581e1"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-dmw7"
server_key: "fc25ba458b56fc45d54ea6b584bc89b6"
server_key_iv: "dddb6033e845d2057e177c8f327792f3"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.301970 10064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.317039 10064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.319026 10064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.319136 10064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.319330 10064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:40.319669 10064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:40.319701 10064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.319723 10064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:40.319778 10064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.325426 10064 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44359
I20260923 22:17:40.325559 10192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44359 every 8 connection(s)
I20260923 22:17:40.325840 10064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:40.329808 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 10064
I20260923 22:17:40.329908 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:40.330099 1541 external_mini_cluster.cc:1428] Setting key d60f906fa17cd66fff648c9fae96a39c
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:40.335273 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.326752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:43491 (local address 127.1.129.126:39913)
0923 22:17:40.326841 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.326843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.327015 (+ 172us) server_negotiation.cc:407] Connection header received
0923 22:17:40.327522 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.327523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.327534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.327590 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.328515 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.328993 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.330038 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.330317 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.330390 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.332783 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.332784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.332784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.332790 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.334027 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.334396 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.334397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.334397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.334449 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.334732 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.334733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.334733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.334838 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.334893 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.335140 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.335202 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 22:17:40.335453 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.326703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:43491)
0923 22:17:40.326892 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.326929 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.327415 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.327644 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.327647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.328242 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.328435 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.328440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.329075 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.329076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.329939 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.329942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.330400 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.330401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.330523 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.331176 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.331186 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.332672 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:40.334099 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.334105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.334107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.334325 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.334513 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.334514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.334514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.334609 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.334901 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:40.334905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:40.335019 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 22:17:40.335097 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 22:17:40.335214 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260923 22:17:40.335987 10193 heartbeater.cc:347] Connected to a master server at 127.1.129.126:39913
I20260923 22:17:40.336095 10193 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:40.336290 10193 heartbeater.cc:511] Master 127.1.129.126:39913 requested a full tablet report, sending...
I20260923 22:17:40.336692 9734 ts_manager.cc:194] Registered new tserver with Master: 3431aa9cdb53452c8dbc6cf23e0581e1 (127.1.129.67:44359)
I20260923 22:17:40.337069 9734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:43491
I20260923 22:17:40.342028 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab.
Entry for principal oryx/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab.
WARNING: no policy specified for oryx/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab.
Entry for principal oryx/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab.
WARNING: no policy specified for oryx/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab.
Entry for principal oryx/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab.
I20260923 22:17:40.417393 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 9796
I20260923 22:17:40.421921 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 9930
I20260923 22:17:40.426155 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 10064
I20260923 22:17:40.430347 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 9702
I20260923 22:17:40.435259 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:39913
--webserver_interface=127.1.129.126
--webserver_port=44881
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:39913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:40.514788 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.514979 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.515000 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.516055 10202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:40.516086 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.516100 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.516113 10202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:40.516124 10202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:40.517458 10202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:39913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=44881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:40.517630 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.517781 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:40.522558 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.522796 10207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.522796 10208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.522815 10202 server_base.cc:1034] running on GCE node
I20260923 22:17:40.523146 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.523458 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.524762 10202 hybrid_clock.cc:648] HybridClock initialized: now 1790201860524746 us; error 46 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.527254 10202 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:40.527608 10202 webserver.cc:460] Webserver started at http://127.1.129.126:44881/ using document root <none> and password file <none>
I20260923 22:17:40.527784 10202 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.527832 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.529048 10202 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:40.529727 10217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.529937 10202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:40.530015 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "17e88933e14a44afb138f091c62ad946"
format_stamp: "Formatted at 2026-09-23 22:17:39 on dist-test-slave-dmw7"
server_key: "ee7e9f39d772ddd6a83706000962f920"
server_key_iv: "ed6fb5b9176756cc21917381ffadde49"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.530301 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.540601 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.542878 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.542981 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.546495 10202 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:39913
I20260923 22:17:40.546548 10269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:39913 every 8 connection(s)
I20260923 22:17:40.546927 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:40.548336 10270 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:17:40.548983 10270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Bootstrap starting.
I20260923 22:17:40.549571 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 10202
I20260923 22:17:40.549835 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:33677
--local_ip_for_outbound_sockets=127.1.129.65
--tserver_master_addrs=127.1.129.126:39913
--webserver_port=36981
--webserver_interface=127.1.129.65
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.129.65
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:17:40.557842 10270 log.cc:824] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:40.559700 10270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:17:40.560015 10270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Bootstrap complete.
I20260923 22:17:40.562247 10270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } }
I20260923 22:17:40.562505 10270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e88933e14a44afb138f091c62ad946, State: Initialized, Role: FOLLOWER
I20260923 22:17:40.562740 10270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } }
I20260923 22:17:40.562836 10270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:40.562896 10270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:40.562947 10270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:17:40.563977 10270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } }
I20260923 22:17:40.564108 10270 leader_election.cc:302] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17e88933e14a44afb138f091c62ad946; no voters:
I20260923 22:17:40.564342 10270 leader_election.cc:288] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 22:17:40.564378 10274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:17:40.564502 10274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [term 2 LEADER]: Becoming Leader. State: Replica: 17e88933e14a44afb138f091c62ad946, State: Running, Role: LEADER
I20260923 22:17:40.564589 10274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } }
I20260923 22:17:40.564618 10270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:40.564885 10275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "17e88933e14a44afb138f091c62ad946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } } }
I20260923 22:17:40.564883 10274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17e88933e14a44afb138f091c62ad946. Latest consensus state: current_term: 2 leader_uuid: "17e88933e14a44afb138f091c62ad946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e88933e14a44afb138f091c62ad946" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39913 } } }
I20260923 22:17:40.565063 10275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:40.565084 10274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:40.565398 10282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:40.565925 10282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:40.566080 10282 catalog_manager.cc:1259] Loaded cluster ID: 90836c0f756d40e886c3a9ecd9fe286f
I20260923 22:17:40.566306 10282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:40.567227 10282 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:40.567422 10282 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 17e88933e14a44afb138f091c62ad946: Loaded TSK: 0
I20260923 22:17:40.567647 10282 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:17:40.645503 10272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.645712 10272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.645735 10272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.647006 10272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.647051 10272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.647078 10272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:40.648520 10272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.65.keytab
--principal=oryx/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:33677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=36981
--tserver_master_addrs=127.1.129.126:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:40.648765 10272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.649031 10272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:40.655043 10299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.654996 10297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.655243 10272 server_base.cc:1034] running on GCE node
W20260923 22:17:40.654992 10296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.655489 10272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.655773 10272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.656937 10272 hybrid_clock.cc:648] HybridClock initialized: now 1790201860656927 us; error 33 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.659745 10272 init.cc:377] Logged in from keytab as oryx/127.1.129.65@KRBTEST.COM (short username oryx)
I20260923 22:17:40.660318 10272 webserver.cc:460] Webserver started at http://127.1.129.65:36981/ using document root <none> and password file <none>
I20260923 22:17:40.660575 10272 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.660661 10272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.662025 10272 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:40.662684 10306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.662962 10272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:40.663061 10272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "789ae423770048a49f1cd090161ccfc6"
format_stamp: "Formatted at 2026-09-23 22:17:39 on dist-test-slave-dmw7"
server_key: "f4ca3e947f3e867ed4bb297a272b4e11"
server_key_iv: "ce2fb608d5f5d4eb31ba787a3898e7f2"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.663501 10272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.679162 10272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.681649 10272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.681823 10272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.682068 10272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:40.682439 10272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:40.682489 10272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.682523 10272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:40.682550 10272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.689839 10272 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:33677
I20260923 22:17:40.689874 10419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:33677 every 8 connection(s)
I20260923 22:17:40.690322 10272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:40.695442 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 10272
I20260923 22:17:40.695747 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:34479
--local_ip_for_outbound_sockets=127.1.129.66
--tserver_master_addrs=127.1.129.126:39913
--webserver_port=40045
--webserver_interface=127.1.129.66
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.129.66
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:40.699446 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.691107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:48377)
0923 22:17:40.691290 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.691334 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.691841 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.692317 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.692322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.692892 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.693077 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.693082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.694207 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.694209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.695110 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.695113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.696005 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.696006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.696124 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.696671 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.696679 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.699228 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
W20260923 22:17:40.699738 10420 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:39913 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.129.126:39913: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
W20260923 22:17:40.700117 10423 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.691216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:48377 (local address 127.1.129.126:39913)
0923 22:17:40.691422 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.691430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.691471 (+ 41us) server_negotiation.cc:407] Connection header received
0923 22:17:40.692053 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.692059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.692111 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.692244 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.693208 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.694091 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.695264 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.695773 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.695911 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.699345 (+ 3434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:48377: BlockingRecv error: recv got EOF from 127.1.129.65:48377 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:40.704505 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.700015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:46657)
0923 22:17:40.700267 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.700278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.700322 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.700484 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.700485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.700609 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.700654 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.700655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.701327 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.701328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.701893 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.701894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.702245 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.702246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.702310 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.702776 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.702781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.704425 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239}
W20260923 22:17:40.704560 10423 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.700062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:46657 (local address 127.1.129.126:39913)
0923 22:17:40.700240 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.700242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.700370 (+ 128us) server_negotiation.cc:407] Connection header received
0923 22:17:40.700395 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.700396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.700405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.700469 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.700733 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.701236 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.701982 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.702161 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.702244 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.704494 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:46657: BlockingRecv error: recv got EOF from 127.1.129.65:46657 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:40.709230 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.705008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:40465)
0923 22:17:40.705091 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.705102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.705145 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.705274 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.705274 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.705397 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.705423 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.705424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.706152 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.706153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.706705 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.706707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.707063 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.707063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.707116 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.707551 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.707556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.709152 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 22:17:40.709281 10423 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.705056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:40465 (local address 127.1.129.126:39913)
0923 22:17:40.705092 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.705094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.705130 (+ 36us) server_negotiation.cc:407] Connection header received
0923 22:17:40.705201 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.705201 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.705211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.705260 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.705578 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.706068 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.706785 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.706976 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.707061 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.709213 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:40465: BlockingRecv error: recv got EOF from 127.1.129.65:40465 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260923 22:17:40.709591 10420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:17:40.785729 10424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.785953 10424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.785979 10424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.787151 10424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.787196 10424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.787231 10424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:40.788617 10424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.66.keytab
--principal=oryx/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:34479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=40045
--tserver_master_addrs=127.1.129.126:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:40.788806 10424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.788991 10424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:40.794183 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.794302 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.794312 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.794440 10424 server_base.cc:1034] running on GCE node
I20260923 22:17:40.794742 10424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.794957 10424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.796108 10424 hybrid_clock.cc:648] HybridClock initialized: now 1790201860796091 us; error 34 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.798789 10424 init.cc:377] Logged in from keytab as oryx/127.1.129.66@KRBTEST.COM (short username oryx)
I20260923 22:17:40.799302 10424 webserver.cc:460] Webserver started at http://127.1.129.66:40045/ using document root <none> and password file <none>
I20260923 22:17:40.799548 10424 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.799634 10424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.801012 10424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:40.801712 10440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.801892 10424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:40.801998 10424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "d870419a8da4499ab0d35428dcb6d16b"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-dmw7"
server_key: "cc17df3513b21fa3b4586507cee509d3"
server_key_iv: "609087953a326d7df9a65254ddc9b9d0"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.802386 10424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.813932 10424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.816300 10424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.816514 10424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.816776 10424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:40.817096 10424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:40.817155 10424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.817196 10424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:40.817225 10424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.823094 10424 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:34479
I20260923 22:17:40.823139 10553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:34479 every 8 connection(s)
I20260923 22:17:40.823524 10424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:40.831377 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 10424
W20260923 22:17:40.831596 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.824447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:47567 (local address 127.1.129.126:39913)
0923 22:17:40.824549 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.824552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.824788 (+ 236us) server_negotiation.cc:407] Connection header received
0923 22:17:40.825345 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.825346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.825361 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.825435 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.826459 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.827051 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.828115 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.828331 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.828429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.831534 (+ 3105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:47567: BlockingRecv error: recv got EOF from 127.1.129.66:47567 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:17:40.831676 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.824397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:47567)
0923 22:17:40.824614 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.824676 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.825222 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.825508 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.825512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.826117 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.826326 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.826332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.827158 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.827159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.827997 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.827999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.828465 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.828466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.828591 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.829177 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.829186 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.831460 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260923 22:17:40.831686 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:44359
--local_ip_for_outbound_sockets=127.1.129.67
--tserver_master_addrs=127.1.129.126:39913
--webserver_port=44437
--webserver_interface=127.1.129.67
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.129.67
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:40.831892 10554 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:39913 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.129.126:39913: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:40.837106 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.832150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:58767)
0923 22:17:40.832189 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.832199 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.832265 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.832388 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.832388 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.832518 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.832562 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.832563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.833510 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.833511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.834230 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.834233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.834578 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.834579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.834681 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.835091 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.835096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.837009 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260923 22:17:40.837201 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.832213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:58767 (local address 127.1.129.126:39913)
0923 22:17:40.832296 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.832298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.832306 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:40.832328 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.832329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.832339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.832387 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.832644 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.833429 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.834322 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.834499 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.834554 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.837105 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:58767: BlockingRecv error: recv got EOF from 127.1.129.66:58767 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:40.842924 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.837587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:51775)
0923 22:17:40.837704 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.837717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.837778 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.837928 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.837928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.838086 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.838126 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.838128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.838883 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.838885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.839760 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.839763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.840115 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.840116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.840180 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.840567 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.840573 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.842845 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260923 22:17:40.843071 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.837653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:51775 (local address 127.1.129.126:39913)
0923 22:17:40.837691 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.837692 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.837809 (+ 117us) server_negotiation.cc:407] Connection header received
0923 22:17:40.837834 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.837834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.837844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.837901 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.838226 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.838794 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.839857 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.840035 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.840092 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.842929 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:51775: BlockingRecv error: recv got EOF from 127.1.129.66:51775 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 22:17:40.843110 10554 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:17:40.913759 10557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.914058 10557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.914098 10557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.915314 10557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.915397 10557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.915443 10557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:40.916901 10557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/krb5kdc/oryx_127.1.129.67.keytab
--principal=oryx/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:44359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=44437
--tserver_master_addrs=127.1.129.126:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:40.917186 10557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.917433 10557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:40.922540 10565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.922559 10562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.922566 10557 server_base.cc:1034] running on GCE node
W20260923 22:17:40.922540 10563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.923054 10557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.923307 10557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.924474 10557 hybrid_clock.cc:648] HybridClock initialized: now 1790201860924458 us; error 26 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.926964 10557 init.cc:377] Logged in from keytab as oryx/127.1.129.67@KRBTEST.COM (short username oryx)
I20260923 22:17:40.927414 10557 webserver.cc:460] Webserver started at http://127.1.129.67:44437/ using document root <none> and password file <none>
I20260923 22:17:40.927645 10557 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.927731 10557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.929080 10557 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:40.929643 10572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.929822 10557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:40.929939 10557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "3431aa9cdb53452c8dbc6cf23e0581e1"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-dmw7"
server_key: "fc25ba458b56fc45d54ea6b584bc89b6"
server_key_iv: "dddb6033e845d2057e177c8f327792f3"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.930308 10557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.943140 10557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.945153 10557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.945327 10557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.945567 10557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:40.945871 10557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:40.945927 10557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.945973 10557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:40.945994 10557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.951829 10557 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44359
I20260923 22:17:40.951870 10685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44359 every 8 connection(s)
I20260923 22:17:40.952196 10557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:40.957082 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 10557
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 22:17:40.960284 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.952968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:38265 (local address 127.1.129.126:39913)
0923 22:17:40.953049 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.953051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.953308 (+ 257us) server_negotiation.cc:407] Connection header received
0923 22:17:40.953848 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.953850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.953864 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.953924 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.954983 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.955529 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.956575 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.956781 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.956894 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.960220 (+ 3326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:38265: BlockingRecv error: recv got EOF from 127.1.129.67:38265 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 22:17:40.960294 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.952951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:38265)
0923 22:17:40.953165 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.953204 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.953680 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.954005 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.954008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.954593 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.954889 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.954894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.955647 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.955648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.956455 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.956457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.956876 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.956877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.956979 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.957715 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.957747 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.960129 (+ 2382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
W20260923 22:17:40.960654 10686 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:39913 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.129.126:39913: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:40.965240 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.957697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.1.129.126:39913)
0923 22:17:40.957772 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.957774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.957801 (+ 27us) server_negotiation.cc:407] Connection header received
0923 22:17:40.957860 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.957861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.957876 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.957941 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.958609 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.959166 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.960021 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.960275 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.960341 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.961841 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.961847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.961847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.961859 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.963781 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.964165 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.964170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.964170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.964284 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.964508 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.964508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.964509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.964736 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.964895 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.965009 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.965118 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 22:17:40.965394 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.960975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:56109 (local address 127.1.129.126:39913)
0923 22:17:40.961029 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.961030 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.961037 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:40.961094 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.961094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.961102 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.961145 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.961440 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.961937 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.962589 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.962932 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.963002 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.965310 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:56109: BlockingRecv error: recv got EOF from 127.1.129.67:56109 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:17:40.965617 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.960910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:56109)
0923 22:17:40.960978 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.960989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.961032 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.961152 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.961153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.961310 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.961369 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.961371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.962020 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.962021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.962510 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.962511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.963017 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.963018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.963069 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.963432 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.963435 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.965237 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:40 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:40.970139 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.966121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:53091)
0923 22:17:40.966299 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.966312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.966358 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.966457 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.966458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.966558 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.966582 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.966583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.967259 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.967260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.967720 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.967721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.968011 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.968012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.968070 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.968417 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.968421 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.970062 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164}
W20260923 22:17:40.970185 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:40.966198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:53091 (local address 127.1.129.126:39913)
0923 22:17:40.966231 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.966233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.966367 (+ 134us) server_negotiation.cc:407] Connection header received
0923 22:17:40.966403 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.966404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.966412 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.966448 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.966621 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.967176 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.967791 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.967961 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.968012 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.970125 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:53091: BlockingRecv error: recv got EOF from 127.1.129.67:53091 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260923 22:17:40.970333 10686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 22:17:41 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:41 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:41.714331 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.709977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:38679)
0923 22:17:41.710133 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.710144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.710198 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.710333 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.710334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.710508 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:41.710553 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.710555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.711236 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.711237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.711780 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.711781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.712100 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.712101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.712140 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.712592 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:41.712597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:41.714264 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
W20260923 22:17:41.714416 10691 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:41.710069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:38679 (local address 127.1.129.126:39913)
0923 22:17:41.710184 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.710187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.710201 (+ 14us) server_negotiation.cc:407] Connection header received
0923 22:17:41.710253 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.710254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.710266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.710314 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.710674 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.711170 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.711855 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.712037 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.712085 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.714365 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:38679: BlockingRecv error: recv got EOF from 127.1.129.65:38679 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
Sep 23 22:17:41 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:41 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:41.848464 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.843471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:51943)
0923 22:17:41.843626 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.843637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.843698 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.843803 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.843804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.843955 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:41.843999 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.844001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.844659 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.844660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.845182 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.845183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.845514 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.845515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.845553 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.845970 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:41.845974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:41.848345 (+ 2371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
W20260923 22:17:41.848534 10691 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:41.843577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:51943 (local address 127.1.129.126:39913)
0923 22:17:41.843623 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.843625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.843705 (+ 80us) server_negotiation.cc:407] Connection header received
0923 22:17:41.843745 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.843745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.843755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.843792 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.844058 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.844588 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.845255 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.845449 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.845500 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.848461 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:51943: BlockingRecv error: recv got EOF from 127.1.129.66:51943 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 23 22:17:41 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:41 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:41.976215 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.970763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:55777)
0923 22:17:41.970912 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.970924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.970989 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.971184 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.971186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.971360 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:41.971408 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.971410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.972442 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.972443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.973003 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.973005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.973552 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.973553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.973603 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.974117 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:41.974122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:41.976148 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
W20260923 22:17:41.976415 10691 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:41.970831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:55777 (local address 127.1.129.126:39913)
0923 22:17:41.970921 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.970923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.971024 (+ 101us) server_negotiation.cc:407] Connection header received
0923 22:17:41.971048 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.971049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.971062 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.971114 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.971496 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.972352 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.973093 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.973448 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.973520 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.976270 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:55777: BlockingRecv error: recv got EOF from 127.1.129.67:55777 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 22:17:42 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:42 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:42.719446 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.714892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:46537)
0923 22:17:42.715035 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:42.715051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:42.715100 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:42.715243 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:42.715244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:42.715418 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:42.715466 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.715469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.716098 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.716098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.716632 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.716634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.716964 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.716964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.717003 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.717463 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:42.717467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:42.719349 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
W20260923 22:17:42.719511 10695 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:42.714981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:46537 (local address 127.1.129.126:39913)
0923 22:17:42.715113 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:42.715115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:42.715140 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:17:42.715160 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:42.715160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:42.715169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:42.715220 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:42.715537 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.716027 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.716708 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.716898 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.716946 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.719436 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:46537: BlockingRecv error: recv got EOF from 127.1.129.65:46537 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
Sep 23 22:17:42 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:42 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:42.853899 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.849124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:51839)
0923 22:17:42.849262 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:42.849274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:42.849341 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:42.849516 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:42.849517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:42.849690 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:42.849736 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.849737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.850662 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.850663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.851258 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.851261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.851590 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.851591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.851632 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.852135 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:42.852141 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:42.853835 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
W20260923 22:17:42.854025 10695 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:42.849235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:51839 (local address 127.1.129.126:39913)
0923 22:17:42.849321 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:42.849324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:42.849336 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:42.849413 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:42.849414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:42.849425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:42.849497 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:42.849837 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.850547 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.851332 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.851514 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.851564 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.853923 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:51839: BlockingRecv error: recv got EOF from 127.1.129.66:51839 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 22:17:42 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:42 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:42.981531 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.976776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:49073)
0923 22:17:42.976922 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:42.976934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:42.976992 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:42.977143 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:42.977143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:42.977317 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:42.977372 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.977374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.978005 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.978006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.978534 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.978536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.978875 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.978876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.978921 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.979372 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:42.979377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:42.981475 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260923 22:17:42.981637 10695 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:42.976937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:49073 (local address 127.1.129.126:39913)
0923 22:17:42.977011 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:42.977013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:42.977023 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:42.977052 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:42.977053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:42.977064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:42.977120 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:42.977447 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.977921 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.978615 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.978809 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.978863 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.981568 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:49073: BlockingRecv error: recv got EOF from 127.1.129.67:49073 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 22:17:43 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:43 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:43.725517 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:43.720059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:43449)
0923 22:17:43.720233 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:43.720244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:43.720300 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:43.720487 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:43.720488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:43.720648 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:43.720695 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:43.720697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.721463 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.721464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:43.722500 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:43.722505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.722928 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.722929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:43.722971 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:43.723519 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:43.723524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:43.725457 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
W20260923 22:17:43.725692 10699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:43.720206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:43449 (local address 127.1.129.126:39913)
0923 22:17:43.720360 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:43.720362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:43.720371 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:43.720394 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:43.720395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:43.720405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:43.720467 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:43.720766 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.721283 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.722590 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.722855 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.722914 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:43.725601 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:43449: BlockingRecv error: recv got EOF from 127.1.129.65:43449 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
Sep 23 22:17:43 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:43 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:43.862846 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:43.855093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:43569)
0923 22:17:43.857115 (+ 2022us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:43.857130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:43.857212 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:43.857413 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:43.857415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:43.857623 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:43.857681 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:43.857683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.858572 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.858574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:43.859503 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:43.859506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.860042 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.860043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:43.860093 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:43.860615 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:43.860620 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:43.862753 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1999,"thread_start_us":76,"threads_started":1}
W20260923 22:17:43.862877 10699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:43.855255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:43569 (local address 127.1.129.126:39913)
0923 22:17:43.855301 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:43.855303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:43.857289 (+ 1986us) server_negotiation.cc:407] Connection header received
0923 22:17:43.857325 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:43.857326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:43.857340 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:43.857401 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:43.857749 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.858471 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.859624 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.859913 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.859981 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:43.862817 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:43569: BlockingRecv error: recv got EOF from 127.1.129.66:43569 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 23 22:17:43 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:43 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:43.986596 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:43.982151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:55467)
0923 22:17:43.982329 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:43.982340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:43.982395 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:43.982584 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:43.982586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:43.982771 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:43.982817 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:43.982819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.983469 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.983470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:43.984000 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:43.984002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.984304 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.984305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:43.984346 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:43.984755 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:43.984760 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:43.986503 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
W20260923 22:17:43.986662 10699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:43.982351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:55467 (local address 127.1.129.126:39913)
0923 22:17:43.982431 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:43.982433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:43.982447 (+ 14us) server_negotiation.cc:407] Connection header received
0923 22:17:43.982469 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:43.982469 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:43.982479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:43.982528 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:43.982909 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.983393 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.984072 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:43.984241 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:43.984289 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:43.986574 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:55467: BlockingRecv error: recv got EOF from 127.1.129.67:55467 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 22:17:44 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:44 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:44.731711 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:44.726046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:44005)
0923 22:17:44.726197 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:44.726209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:44.726323 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:44.726477 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:44.726478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:44.726713 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:44.726773 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:44.726776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.727908 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.727909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:44.728493 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:44.728495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.729017 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.729018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:44.729074 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.729552 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:44.729557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:44.731628 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
W20260923 22:17:44.731876 10703 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:44.726098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:44005 (local address 127.1.129.126:39913)
0923 22:17:44.726217 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:44.726220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:44.726303 (+ 83us) server_negotiation.cc:407] Connection header received
0923 22:17:44.726394 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:44.726395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:44.726407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:44.726472 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:44.726855 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.727786 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.728561 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.728931 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.728990 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.731763 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:44005: BlockingRecv error: recv got EOF from 127.1.129.65:44005 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
Sep 23 22:17:44 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:44 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:44.868661 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:44.863385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:33705)
0923 22:17:44.863588 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:44.863601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:44.863701 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:44.863866 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:44.863868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:44.864065 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:44.864109 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:44.864111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.864911 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.864912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:44.865593 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:44.865596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.866061 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.866062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:44.866120 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.866705 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:44.866710 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:44.868564 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
W20260923 22:17:44.868712 10703 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:44.863506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:33705 (local address 127.1.129.126:39913)
0923 22:17:44.863585 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:44.863587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:44.863680 (+ 93us) server_negotiation.cc:407] Connection header received
0923 22:17:44.863772 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:44.863773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:44.863791 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:44.863872 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:44.864196 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.864811 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.865689 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.865971 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.866060 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.868649 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:33705: BlockingRecv error: recv got EOF from 127.1.129.66:33705 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 22:17:44 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:44 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:44.994988 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:44.987254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:34153)
0923 22:17:44.987415 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:44.987428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:44.987515 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:44.987690 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:44.987691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:44.987877 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:44.987923 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:44.987925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.988587 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.988588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:44.989109 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:44.989111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.989466 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.989468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:44.989530 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.993007 (+ 3477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:44.993012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:44.994890 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
W20260923 22:17:44.995038 10703 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:44.987325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:34153 (local address 127.1.129.126:39913)
0923 22:17:44.987405 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:44.987407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:44.987485 (+ 78us) server_negotiation.cc:407] Connection header received
0923 22:17:44.987592 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:44.987593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:44.987609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:44.987682 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:44.988006 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.988494 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.989194 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.989384 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.989456 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.994971 (+ 5515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:34153: BlockingRecv error: recv got EOF from 127.1.129.67:34153 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 22:17:45 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.736941 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.732203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:36781)
0923 22:17:45.732368 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.732378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.732429 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.732593 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.732594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.732750 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.732798 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.732800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.733537 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.733537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.734048 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.734050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.734428 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.734429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.734471 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.734993 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.735002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.736842 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
W20260923 22:17:45.736999 10707 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.732274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:36781 (local address 127.1.129.126:39913)
0923 22:17:45.732404 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.732406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.732417 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:45.732474 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.732474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.732486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.732582 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.732867 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.733464 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.734121 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.734363 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.734416 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.736927 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:36781: BlockingRecv error: recv got EOF from 127.1.129.65:36781 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
Sep 23 22:17:45 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.873656 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.869148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:35661)
0923 22:17:45.869286 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.869303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.869367 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.869505 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.869505 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.869665 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.869714 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.869716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.870395 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.870396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.870978 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.870981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.871346 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.871347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.871384 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.871855 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.871860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.873596 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
W20260923 22:17:45.873737 10707 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.869271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:35661 (local address 127.1.129.126:39913)
0923 22:17:45.869320 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.869322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.869356 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:17:45.869413 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.869414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.869425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.869486 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.869812 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.870315 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.871078 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.871270 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.871334 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.873691 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:35661: BlockingRecv error: recv got EOF from 127.1.129.66:35661 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 23 22:17:45 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:46.000059 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.995534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:42945)
0923 22:17:45.995656 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.995668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.995727 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.995865 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.995865 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.996022 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.996068 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.996070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.996714 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.996715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.997235 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.997236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.997624 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.997625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.997663 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.998122 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.998127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:46.000002 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260923 22:17:46.000209 10707 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.995612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:42945 (local address 127.1.129.126:39913)
0923 22:17:45.995716 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.995718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.995736 (+ 18us) server_negotiation.cc:407] Connection header received
0923 22:17:45.995774 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.995774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.995785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.995834 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.996136 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.996627 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.997345 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.997549 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.997617 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.000118 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:42945: BlockingRecv error: recv got EOF from 127.1.129.67:42945 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 22:17:46 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:46 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:46.742133 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:46.737435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:51647)
0923 22:17:46.737596 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:46.737607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:46.737672 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:46.737819 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:46.737820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:46.737981 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:46.738028 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.738030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.738724 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.738725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.739242 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.739244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.739576 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.739577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.739617 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.740061 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:46.740065 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:46.742046 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
W20260923 22:17:46.742235 10711 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:46.737525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:51647 (local address 127.1.129.126:39913)
0923 22:17:46.737624 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:46.737626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:46.737655 (+ 29us) server_negotiation.cc:407] Connection header received
0923 22:17:46.737732 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:46.737733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:46.737743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:46.737802 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:46.738094 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.738586 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.739317 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.739509 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.739558 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.742151 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:51647: BlockingRecv error: recv got EOF from 127.1.129.65:51647 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":50,"threads_started":1}
Sep 23 22:17:46 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:46 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:46.878891 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:46.874254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:33595)
0923 22:17:46.874404 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:46.874420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:46.874484 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:46.874596 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:46.874597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:46.874836 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:46.874885 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.874888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.875519 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.875520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.876048 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.876049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.876366 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.876367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.876408 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.876861 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:46.876865 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:46.878832 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
W20260923 22:17:46.879052 10711 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:46.874310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:33595 (local address 127.1.129.126:39913)
0923 22:17:46.874397 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:46.874399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:46.874483 (+ 84us) server_negotiation.cc:407] Connection header received
0923 22:17:46.874531 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:46.874531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:46.874541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:46.874591 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:46.874957 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.875446 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.876124 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.876299 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.876358 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.878973 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:33595: BlockingRecv error: recv got EOF from 127.1.129.66:33595 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 22:17:47 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:47 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:47.005332 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:47.000625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:42747)
0923 22:17:47.000783 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:47.000795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:47.000876 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:47.001019 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:47.001020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:47.001175 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:47.001222 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.001224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.001994 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.001996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.002549 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.002551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.002933 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.002934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.003000 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.003444 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:47.003448 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:47.005237 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
W20260923 22:17:47.005390 10711 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:47.000730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:42747 (local address 127.1.129.126:39913)
0923 22:17:47.000803 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:47.000805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:47.000882 (+ 77us) server_negotiation.cc:407] Connection header received
0923 22:17:47.000923 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:47.000924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:47.000934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:47.000996 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:47.001308 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.001895 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.002664 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.002854 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.002935 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.005312 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:42747: BlockingRecv error: recv got EOF from 127.1.129.67:42747 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 22:17:47 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:47 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:47.747133 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:47.742673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:43537)
0923 22:17:47.742825 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:47.742835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:47.742903 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:47.743036 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:47.743036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:47.743193 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:47.743240 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.743242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.743895 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.743895 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.744407 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.744408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.744726 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.744727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.744765 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.745204 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:47.745208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:47.747076 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
W20260923 22:17:47.747344 10715 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:47.742734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:43537 (local address 127.1.129.126:39913)
0923 22:17:47.742853 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:47.742855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:47.742905 (+ 50us) server_negotiation.cc:407] Connection header received
0923 22:17:47.742957 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:47.742957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:47.742968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:47.743016 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:47.743313 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.743813 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.744478 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.744663 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.744711 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.747247 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:43537: BlockingRecv error: recv got EOF from 127.1.129.65:43537 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
Sep 23 22:17:47 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:47 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:47.884116 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:47.879363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:45145)
0923 22:17:47.879507 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:47.879518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:47.879577 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:47.879690 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:47.879691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:47.879852 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:47.879899 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.879900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.880518 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.880519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.881275 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.881278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.881596 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.881596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.881634 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.882058 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:47.882062 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:47.884061 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260923 22:17:47.884196 10715 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:47.879436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:45145 (local address 127.1.129.126:39913)
0923 22:17:47.879487 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:47.879489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:47.879585 (+ 96us) server_negotiation.cc:407] Connection header received
0923 22:17:47.879628 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:47.879628 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:47.879638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:47.879685 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:47.879958 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.880450 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.881346 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.881533 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.881586 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.884150 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:45145: BlockingRecv error: recv got EOF from 127.1.129.66:45145 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 23 22:17:48 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:48 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:48.010690 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:48.005941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:45389)
0923 22:17:48.006126 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:48.006137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:48.006205 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:48.006363 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:48.006364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:48.006523 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:48.006571 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.006573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.007225 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.007226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.007762 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.007764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.008144 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.008145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.008195 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.008705 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:48.008709 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:48.010564 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
W20260923 22:17:48.010797 10715 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:48.006064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:45389 (local address 127.1.129.126:39913)
0923 22:17:48.006140 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:48.006142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:48.006188 (+ 46us) server_negotiation.cc:407] Connection header received
0923 22:17:48.006273 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:48.006274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:48.006285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:48.006336 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:48.006659 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.007138 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.007866 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.008053 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.008112 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.010720 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:45389: BlockingRecv error: recv got EOF from 127.1.129.67:45389 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 22:17:48 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:48 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:48.752300 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:48.747629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:58159)
0923 22:17:48.747877 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:48.747891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:48.747950 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:48.748089 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:48.748090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:48.748255 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:48.748306 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.748308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.748959 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.748960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.749482 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.749483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.749836 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.749837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.749876 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.750355 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:48.750360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:48.752189 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":192,"threads_started":1}
W20260923 22:17:48.752349 10719 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:48.747740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:58159 (local address 127.1.129.126:39913)
0923 22:17:48.747880 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:48.747882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:48.747937 (+ 55us) server_negotiation.cc:407] Connection header received
0923 22:17:48.748004 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:48.748005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:48.748015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:48.748065 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:48.748390 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.748889 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.749572 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.749765 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.749821 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.752277 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:58159: BlockingRecv error: recv got EOF from 127.1.129.65:58159 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Sep 23 22:17:48 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:48 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:48.890076 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:48.884814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:44965)
0923 22:17:48.884945 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:48.884958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:48.885059 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:48.885213 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:48.885213 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:48.885403 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:48.885450 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.885452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.886254 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.886255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.886860 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.886863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.887252 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.887253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.887315 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.887829 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:48.887834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:48.890007 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260923 22:17:48.890146 10719 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:48.884916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:44965 (local address 127.1.129.126:39913)
0923 22:17:48.884966 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:48.884968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:48.885041 (+ 73us) server_negotiation.cc:407] Connection header received
0923 22:17:48.885146 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:48.885147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:48.885159 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:48.885208 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:48.885536 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.886165 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.886950 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.887169 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.887243 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.890099 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:44965: BlockingRecv error: recv got EOF from 127.1.129.66:44965 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 23 22:17:49 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:49 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:49.016024 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:49.011340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:36191)
0923 22:17:49.011519 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:49.011532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:49.011627 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:49.011776 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:49.011777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:49.011961 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:49.012007 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.012009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.012685 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.012686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.013213 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.013215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.013625 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.013626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.013685 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.014119 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:49.014123 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:49.015954 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
W20260923 22:17:49.016228 10719 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:49.011424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:36191 (local address 127.1.129.126:39913)
0923 22:17:49.011523 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:49.011526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:49.011608 (+ 82us) server_negotiation.cc:407] Connection header received
0923 22:17:49.011711 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:49.011712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:49.011725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:49.011772 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:49.012094 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.012595 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.013352 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.013547 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.013627 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.016143 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:36191: BlockingRecv error: recv got EOF from 127.1.129.67:36191 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 23 22:17:49 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:49 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:49.757365 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:49.752887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:58827)
0923 22:17:49.753004 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:49.753015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:49.753066 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:49.753173 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:49.753174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:49.753321 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:49.753374 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.753375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.753969 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.753970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.754473 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.754474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.754935 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.754935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.754975 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.755445 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:49.755450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:49.757299 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
W20260923 22:17:49.757586 10723 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:49.752921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:58827 (local address 127.1.129.126:39913)
0923 22:17:49.753052 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:49.753054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:49.753062 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:49.753106 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:49.753107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:49.753117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:49.753170 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:49.753427 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.753904 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.754548 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.754826 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.754935 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.757501 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:58827: BlockingRecv error: recv got EOF from 127.1.129.65:58827 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
Sep 23 22:17:49 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:49 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:49.895524 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:49.890687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:55047)
0923 22:17:49.890855 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:49.890868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:49.890934 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:49.891064 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:49.891064 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:49.891226 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:49.891273 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.891275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.892169 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.892170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.892693 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.892694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.893032 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.893033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.893071 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.893528 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:49.893532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:49.895464 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
W20260923 22:17:49.895628 10723 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:49.890745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:55047 (local address 127.1.129.126:39913)
0923 22:17:49.890855 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:49.890857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:49.890924 (+ 67us) server_negotiation.cc:407] Connection header received
0923 22:17:49.890983 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:49.890984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:49.890994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:49.891044 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:49.891339 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.892103 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.892771 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.892961 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.893016 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.895560 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:55047: BlockingRecv error: recv got EOF from 127.1.129.66:55047 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 23 22:17:50 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:50 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:50.025702 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:50.016567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:46045)
0923 22:17:50.016685 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:50.016697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:50.016763 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:50.016924 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:50.016925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:50.017131 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:50.017176 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.017177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.017829 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.017830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.018376 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.018377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.018802 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.018804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.018870 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.023700 (+ 4830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:50.023705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:50.025609 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
W20260923 22:17:50.025750 10723 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:50.016678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:46045 (local address 127.1.129.126:39913)
0923 22:17:50.016729 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:50.016731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:50.016739 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:50.016826 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:50.016827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:50.016841 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:50.016918 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:50.017257 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.017743 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.018463 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.018717 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.018800 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.025686 (+ 6886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:46045: BlockingRecv error: recv got EOF from 127.1.129.67:46045 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 23 22:17:50 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:50 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:50.762387 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:50.757846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:56195)
0923 22:17:50.758010 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:50.758020 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:50.758090 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:50.758220 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:50.758221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:50.758416 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:50.758482 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.758484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.759296 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.759297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.759814 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.759816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.760125 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.760125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.760162 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.760583 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:50.760587 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:50.762317 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
W20260923 22:17:50.762600 10727 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:50.757949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:56195 (local address 127.1.129.126:39913)
0923 22:17:50.758066 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:50.758067 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:50.758077 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:50.758145 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:50.758145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:50.758155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:50.758203 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:50.758557 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.759227 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.759884 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.760064 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.760112 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.762532 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:56195: BlockingRecv error: recv got EOF from 127.1.129.65:56195 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":48,"threads_started":1}
Sep 23 22:17:50 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:50 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:50.901041 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:50.896040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:55111)
0923 22:17:50.896242 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:50.896255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:50.896346 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:50.896499 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:50.896500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:50.896681 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:50.896728 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.896730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.897406 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.897407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.897943 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.897945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.898328 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.898329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.898384 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.898952 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:50.898964 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:50.900936 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
W20260923 22:17:50.901077 10727 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:50.896151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:55111 (local address 127.1.129.126:39913)
0923 22:17:50.896227 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:50.896229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:50.896329 (+ 100us) server_negotiation.cc:407] Connection header received
0923 22:17:50.896403 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:50.896404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:50.896415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:50.896493 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:50.896812 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.897312 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.898043 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.898241 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.898326 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.901014 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:55111: BlockingRecv error: recv got EOF from 127.1.129.66:55111 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 22:17:51 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:51 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:51.031096 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:51.026264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:40521)
0923 22:17:51.026470 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:51.026489 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:51.026551 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:51.026721 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:51.026722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:51.026955 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:51.027001 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.027003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.027639 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.027639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.028148 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.028150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.028510 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.028511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.028561 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.029038 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:51.029043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:51.031029 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
W20260923 22:17:51.031266 10727 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:51.026424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:40521 (local address 127.1.129.126:39913)
0923 22:17:51.026496 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:51.026498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:51.026536 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:17:51.026595 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:51.026596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:51.026607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:51.026687 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:51.027075 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.027568 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.028218 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.028395 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.028482 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.031165 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:40521: BlockingRecv error: recv got EOF from 127.1.129.67:40521 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 22:17:51 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:51 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:51.767522 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:51.762910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:44683)
0923 22:17:51.763042 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:51.763053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:51.763100 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:51.763242 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:51.763242 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:51.763405 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:51.763450 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.763451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.764055 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.764056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.764723 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.764726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.765041 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.765042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.765084 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.765628 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:51.765633 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:51.767456 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
W20260923 22:17:51.767647 10731 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:51.762981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:44683 (local address 127.1.129.126:39913)
0923 22:17:51.763100 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:51.763102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:51.763143 (+ 41us) server_negotiation.cc:407] Connection header received
0923 22:17:51.763163 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:51.763164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:51.763173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:51.763221 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:51.763509 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.763989 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.764793 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.764978 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.765029 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.767542 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:44683: BlockingRecv error: recv got EOF from 127.1.129.65:44683 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
Sep 23 22:17:51 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:51 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:51.906801 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:51.901563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:60861)
0923 22:17:51.901788 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:51.901806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:51.901871 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:51.902008 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:51.902009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:51.902169 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:51.902212 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.902214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.902964 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.902965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.903560 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.903562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.903941 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.903942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.903993 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.904496 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:51.904501 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:51.906727 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":174,"threads_started":1}
W20260923 22:17:51.906895 10731 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:51.901653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:60861 (local address 127.1.129.126:39913)
0923 22:17:51.901724 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:51.901726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:51.901885 (+ 159us) server_negotiation.cc:407] Connection header received
0923 22:17:51.901919 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:51.901920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:51.901931 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:51.901992 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:51.902286 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.902870 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.903647 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.903850 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.903905 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.906821 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:60861: BlockingRecv error: recv got EOF from 127.1.129.66:60861 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 22:17:52 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:52 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:52.036610 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:52.031686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:41707)
0923 22:17:52.031887 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:52.031899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:52.031997 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:52.032153 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:52.032155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:52.032434 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:52.032478 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.032481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.033320 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.033321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.033887 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.033889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.034267 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.034268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.034334 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.034801 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:52.034805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:52.036543 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
W20260923 22:17:52.036705 10731 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:52.031780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:41707 (local address 127.1.129.126:39913)
0923 22:17:52.031876 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:52.031879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:52.031999 (+ 120us) server_negotiation.cc:407] Connection header received
0923 22:17:52.032077 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:52.032078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:52.032092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:52.032147 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:52.032563 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.033199 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.033990 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.034194 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.034269 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.036637 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:41707: BlockingRecv error: recv got EOF from 127.1.129.67:41707 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 22:17:52 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:52 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:52.772496 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:52.767968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:34579)
0923 22:17:52.768131 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:52.768143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:52.768250 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:52.768434 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:52.768435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:52.768593 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:52.768637 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.768639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.769232 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.769232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.769729 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.769730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.770070 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.770070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.770108 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.770598 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:52.770603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:52.772442 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260923 22:17:52.772598 10735 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:52.768072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:34579 (local address 127.1.129.126:39913)
0923 22:17:52.768207 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:52.768209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:52.768217 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:52.768316 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:52.768317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:52.768367 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:52.768428 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:52.768701 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.769170 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.769798 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.770010 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.770058 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.772527 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:34579: BlockingRecv error: recv got EOF from 127.1.129.65:34579 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
Sep 23 22:17:52 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:52 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:52.911955 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:52.907298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:36623)
0923 22:17:52.907480 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:52.907493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:52.907582 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:52.907730 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:52.907731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:52.907923 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:52.907965 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.907967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.908640 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.908641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.909167 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.909169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.909545 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.909546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.909608 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.910052 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:52.910056 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:52.911864 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
W20260923 22:17:52.911988 10735 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:52.907407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:36623 (local address 127.1.129.126:39913)
0923 22:17:52.907486 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:52.907488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:52.907561 (+ 73us) server_negotiation.cc:407] Connection header received
0923 22:17:52.907668 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:52.907669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:52.907680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:52.907727 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:52.908052 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.908550 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.909250 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.909460 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.909533 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.911930 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:36623: BlockingRecv error: recv got EOF from 127.1.129.66:36623 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 22:17:53 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:53 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:53.041286 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:53.037193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:40619)
0923 22:17:53.037348 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:53.037359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:53.037432 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:53.037535 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:53.037536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:53.037707 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:53.037749 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.037751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.038333 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.038334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.038910 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.038913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.039213 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.039214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.039252 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.039651 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:53.039655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:53.041229 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
W20260923 22:17:53.041384 10735 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:53.037287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:40619 (local address 127.1.129.126:39913)
0923 22:17:53.037335 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:53.037337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:53.037435 (+ 98us) server_negotiation.cc:407] Connection header received
0923 22:17:53.037472 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:53.037472 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:53.037482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:53.037523 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:53.037803 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.038271 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.038981 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.039153 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.039201 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.041311 (+ 2110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:40619: BlockingRecv error: recv got EOF from 127.1.129.67:40619 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 23 22:17:53 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:53 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:53.777688 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:53.772977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:57919)
0923 22:17:53.773152 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:53.773162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:53.773219 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:53.773327 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:53.773328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:53.773487 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:53.773531 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.773533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.774195 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.774195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.774799 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.774802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.775142 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.775143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.775181 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.775678 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:53.775683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:53.777578 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
W20260923 22:17:53.777768 10739 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:53.773065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:57919 (local address 127.1.129.126:39913)
0923 22:17:53.773170 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:53.773172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:53.773230 (+ 58us) server_negotiation.cc:407] Connection header received
0923 22:17:53.773260 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:53.773261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:53.773270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:53.773324 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:53.773591 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.774129 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.774872 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.775082 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.775131 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.777692 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:57919: BlockingRecv error: recv got EOF from 127.1.129.65:57919 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
Sep 23 22:17:53 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:53 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:53.916793 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:53.912405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:55091)
0923 22:17:53.912549 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:53.912562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:53.912623 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:53.912751 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:53.912751 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:53.912914 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:53.912958 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.912960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.913585 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.913586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.914122 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.914124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.914458 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.914459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.914503 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.915010 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:53.915015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:53.916728 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
W20260923 22:17:53.916882 10739 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:53.912522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:55091 (local address 127.1.129.126:39913)
0923 22:17:53.912565 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:53.912568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:53.912618 (+ 50us) server_negotiation.cc:407] Connection header received
0923 22:17:53.912668 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:53.912669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:53.912679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:53.912725 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:53.913021 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.913516 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.914195 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.914395 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.914447 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.916816 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:55091: BlockingRecv error: recv got EOF from 127.1.129.66:55091 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 23 22:17:54 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:54 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:54.046697 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:54.041908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:54751)
0923 22:17:54.042120 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:54.042133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:54.042263 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:54.042425 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:54.042426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:54.042610 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:54.042682 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.042684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.043412 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.043413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.043948 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.043950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.044313 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.044314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.044384 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.044827 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:54.044831 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:54.046574 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
W20260923 22:17:54.046859 10739 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:54.042009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:54751 (local address 127.1.129.126:39913)
0923 22:17:54.042109 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:54.042111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:54.042242 (+ 131us) server_negotiation.cc:407] Connection header received
0923 22:17:54.042325 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:54.042326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:54.042347 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:54.042416 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:54.042770 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.043329 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.044037 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.044239 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.044315 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.046721 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:54751: BlockingRecv error: recv got EOF from 127.1.129.67:54751 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 22:17:54 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:54 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:54.783007 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:54.778193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:58653)
0923 22:17:54.778362 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:54.778375 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:54.778480 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:54.778676 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:54.778678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:54.778857 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:54.778904 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.778906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.779611 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.779612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.780140 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.780142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.780545 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.780546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.780595 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.781105 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:54.781110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:54.782948 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
W20260923 22:17:54.783121 10743 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:54.778249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:58653 (local address 127.1.129.126:39913)
0923 22:17:54.778402 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:54.778404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:54.778416 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:54.778554 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:54.778555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:54.778565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:54.778620 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:54.778995 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.779522 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.780228 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.780443 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.780499 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.783044 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:58653: BlockingRecv error: recv got EOF from 127.1.129.65:58653 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
Sep 23 22:17:54 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:54 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:54.921921 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:54.917404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:41639)
0923 22:17:54.917571 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:54.917582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:54.917640 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:54.917789 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:54.917790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:54.917953 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:54.917996 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.917997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.918729 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.918731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.919262 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.919264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.919594 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.919594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.919632 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.920143 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:54.920147 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:54.921860 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
W20260923 22:17:54.922086 10743 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:54.917472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:41639 (local address 127.1.129.126:39913)
0923 22:17:54.917557 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:54.917559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:54.917669 (+ 110us) server_negotiation.cc:407] Connection header received
0923 22:17:54.917699 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:54.917700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:54.917712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:54.917774 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:54.918093 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.918604 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.919337 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.919531 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.919584 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.921988 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:41639: BlockingRecv error: recv got EOF from 127.1.129.66:41639 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 22:17:55 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:55 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.67@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:55.058418 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:55.047157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.67:36717)
0923 22:17:55.047390 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:55.047403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:55.047490 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:55.047616 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:55.047617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:55.047817 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:55.047860 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.047862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.048556 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.048557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.049102 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.049104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.049484 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.049485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.049545 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.056469 (+ 6924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:55.056475 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:55.058319 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":137,"threads_started":1}
W20260923 22:17:55.058480 10743 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:55.047254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:36717 (local address 127.1.129.126:39913)
0923 22:17:55.047391 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:55.047393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:55.047471 (+ 78us) server_negotiation.cc:407] Connection header received
0923 22:17:55.047546 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:55.047546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:55.047560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:55.047612 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:55.047945 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.048465 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.049195 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.049408 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.049487 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.058418 (+ 8931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.67:36717: BlockingRecv error: recv got EOF from 127.1.129.67:36717 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 23 22:17:55 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:55 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.65@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:55.788444 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:55.783675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.65:46525)
0923 22:17:55.783815 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:55.783831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:55.783890 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:55.784034 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:55.784034 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:55.784191 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:55.784240 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.784241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.784909 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.784910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.785688 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.785690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.786065 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.786066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.786101 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.786543 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:55.786547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:55.788379 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260923 22:17:55.788540 10747 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:55.783739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:46525 (local address 127.1.129.126:39913)
0923 22:17:55.783902 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:55.783903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:55.783914 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:55.783942 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:55.783942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:55.783953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:55.784022 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:55.784307 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.784799 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.785764 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.785998 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.786057 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.788471 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.65:46525: BlockingRecv error: recv got EOF from 127.1.129.65:46525 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
Sep 23 22:17:55 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:55 dist-test-slave-dmw7 krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.129.66@KRBTEST.COM for oryx/127.1.129.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:55.927304 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:55.922581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39913 (local address 127.1.129.66:54475)
0923 22:17:55.922739 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:55.922751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:55.922819 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:55.922980 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:55.922981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:55.923156 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:55.923200 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.923202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.923866 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.923867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.924391 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.924393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.924745 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.924746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.924849 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.925364 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:55.925369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:55.927190 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.129.126:39913: Server oryx/127.1.129.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
W20260923 22:17:55.927351 10747 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:55.922710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:54475 (local address 127.1.129.126:39913)
0923 22:17:55.922758 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:55.922760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:55.922799 (+ 39us) server_negotiation.cc:407] Connection header received
0923 22:17:55.922884 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:55.922885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:55.922897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:55.922973 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:55.923277 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.923780 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.924480 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.924666 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.924736 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.927273 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.129.66:54475: BlockingRecv error: recv got EOF from 127.1.129.66:54475 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:55.967628 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 10272
I20260923 22:17:55.971918 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 10424
I20260923 22:17:55.976115 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 10557
I20260923 22:17:55.980208 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 10202
2026-09-23T22:17:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17444 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T22:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:56Z Disabled control of system clock
I20260923 22:17:56.007306 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34777
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 22:17:56.086720 10752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:56.086925 10752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:56.086944 10752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:56.088060 10752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:56.088095 10752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:56.088109 10752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:56.088121 10752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:56.088133 10752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:56.089380 10752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:34777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:56.089562 10752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:56.089797 10752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:56.095335 10760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.095400 10757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.095436 10752 server_base.cc:1034] running on GCE node
W20260923 22:17:56.095448 10758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.095814 10752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:56.096107 10752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:56.097247 10752 hybrid_clock.cc:648] HybridClock initialized: now 1790201876097235 us; error 28 us; skew 500 ppm
I20260923 22:17:56.098352 10752 webserver.cc:460] Webserver started at http://127.1.129.126:34211/ using document root <none> and password file <none>
I20260923 22:17:56.098546 10752 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:56.098585 10752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:56.098733 10752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:56.099730 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "f9d5e9a4358c405db0052241cfc37f36"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "43ca8825d1b8ded262b46a84aaba12a8"
server_key_iv: "688906d8e888ad445266d9e6be13ff98"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.100030 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "f9d5e9a4358c405db0052241cfc37f36"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "43ca8825d1b8ded262b46a84aaba12a8"
server_key_iv: "688906d8e888ad445266d9e6be13ff98"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.101732 10752 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 22:17:56.103214 10766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.103403 10752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:56.103488 10752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "f9d5e9a4358c405db0052241cfc37f36"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "43ca8825d1b8ded262b46a84aaba12a8"
server_key_iv: "688906d8e888ad445266d9e6be13ff98"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.103562 10752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:56.112764 10752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:56.113055 10752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:56.113204 10752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:56.117110 10752 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:34777
I20260923 22:17:56.117141 10818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:34777 every 8 connection(s)
I20260923 22:17:56.117470 10752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:56.118112 10819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:56.120604 10819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36: Bootstrap starting.
I20260923 22:17:56.121126 10819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:56.121413 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 10752
I20260923 22:17:56.121457 10819 log.cc:824] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:56.121497 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:56.121701 1541 external_mini_cluster.cc:1428] Setting key 69e0a20ffb92f4f8489e40ae80903882
I20260923 22:17:56.122210 10819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36: No bootstrap required, opened a new log
I20260923 22:17:56.123870 10819 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d5e9a4358c405db0052241cfc37f36" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34777 } }
I20260923 22:17:56.123997 10819 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:56.124019 10819 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d5e9a4358c405db0052241cfc37f36, State: Initialized, Role: FOLLOWER
I20260923 22:17:56.124189 10819 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d5e9a4358c405db0052241cfc37f36" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34777 } }
I20260923 22:17:56.124281 10819 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:56.124317 10819 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:56.124388 10819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:56.125168 10819 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d5e9a4358c405db0052241cfc37f36" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34777 } }
I20260923 22:17:56.125281 10819 leader_election.cc:302] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9d5e9a4358c405db0052241cfc37f36; no voters:
I20260923 22:17:56.125443 10819 leader_election.cc:288] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:56.125490 10824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:56.125628 10824 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [term 1 LEADER]: Becoming Leader. State: Replica: f9d5e9a4358c405db0052241cfc37f36, State: Running, Role: LEADER
I20260923 22:17:56.125684 10819 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:56.125710 10824 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d5e9a4358c405db0052241cfc37f36" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34777 } }
I20260923 22:17:56.126029 10825 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9d5e9a4358c405db0052241cfc37f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d5e9a4358c405db0052241cfc37f36" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34777 } } }
I20260923 22:17:56.126157 10826 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9d5e9a4358c405db0052241cfc37f36. Latest consensus state: current_term: 1 leader_uuid: "f9d5e9a4358c405db0052241cfc37f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d5e9a4358c405db0052241cfc37f36" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34777 } } }
I20260923 22:17:56.126490 10826 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:56.126353 10825 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36 [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:56.127511 10839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:56.127573 10839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:56.127750 10838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:56.128258 10838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:56.129164 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.122217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.1.129.126:34777)
0923 22:17:56.122415 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.122423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.122447 (+ 24us) server_negotiation.cc:407] Connection header received
0923 22:17:56.123861 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.123867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.123939 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.124090 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.124340 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.125313 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.126473 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.127169 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.127282 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.128333 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:56.128338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:56.128338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:56.128546 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:56.128589 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:56.128603 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:56.128661 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:56.128875 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 22:17:56.128953 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260923 22:17:56.129729 10838 catalog_manager.cc:1347] Generated new cluster ID: 992a0eb73b64499eabdb75c211f88652
I20260923 22:17:56.129797 10838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:56.149768 10838 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:56.150338 10838 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:56.163077 10838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9d5e9a4358c405db0052241cfc37f36: Generated new TSK 0
I20260923 22:17:56.163359 10838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:56.169785 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34777
--builtin_ntp_servers=127.1.129.84:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 22:17:56.253651 10843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:56.253871 10843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:56.253894 10843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:56.255141 10843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:56.255182 10843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:56.255209 10843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:56.256498 10843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:56.256685 10843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:56.256945 10843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:56.261968 10851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.262019 10849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.261977 10848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.262336 10843 server_base.cc:1034] running on GCE node
I20260923 22:17:56.262562 10843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:56.262825 10843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:56.263970 10843 hybrid_clock.cc:648] HybridClock initialized: now 1790201876263964 us; error 32 us; skew 500 ppm
I20260923 22:17:56.265311 10843 webserver.cc:460] Webserver started at http://127.1.129.65:34023/ using document root <none> and password file <none>
I20260923 22:17:56.265578 10843 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:56.265626 10843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:56.265797 10843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:56.266939 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "4c8e47316ba24641b751aa0ccbe6ba78"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "59df8590b4656f53f8730b1a1940c224"
server_key_iv: "42a0401b97542fd679766d7414c0a450"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.267285 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "4c8e47316ba24641b751aa0ccbe6ba78"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "59df8590b4656f53f8730b1a1940c224"
server_key_iv: "42a0401b97542fd679766d7414c0a450"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.268625 10843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:56.269390 10857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.269564 10843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:56.269665 10843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "4c8e47316ba24641b751aa0ccbe6ba78"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "59df8590b4656f53f8730b1a1940c224"
server_key_iv: "42a0401b97542fd679766d7414c0a450"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.269742 10843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:56.300220 10843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:56.300557 10843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:56.300711 10843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:56.300964 10843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:56.301303 10843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:56.301357 10843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.301389 10843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:56.301440 10843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.307345 10843 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:33501
I20260923 22:17:56.307458 10970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:33501 every 8 connection(s)
I20260923 22:17:56.307753 10843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:56.313035 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.308471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:46283 (local address 127.1.129.126:34777)
0923 22:17:56.308540 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.308542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.308799 (+ 257us) server_negotiation.cc:407] Connection header received
0923 22:17:56.309329 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.309330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.309341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.309397 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.310118 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.310615 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.311619 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.311826 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.311902 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.312723 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:56.312725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:56.312725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:56.312732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:56.312747 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:56.312752 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:56.312823 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:56.312926 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 22:17:56.312957 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:56.313246 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.308477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34777 (local address 127.1.129.65:46283)
0923 22:17:56.308667 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:56.308699 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:56.309215 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:56.309484 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:56.309488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:56.309505 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:56.310004 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.310009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.310730 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.310731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.311512 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.311514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.311910 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.311911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.312058 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.312571 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:56.312571 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:56.312599 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:56.312621 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:56.312835 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:56.312840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:56.312843 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:56.312930 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:17:56.313027 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260923 22:17:56.313763 10971 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34777
I20260923 22:17:56.313843 10971 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:56.313978 10971 heartbeater.cc:511] Master 127.1.129.126:34777 requested a full tablet report, sending...
I20260923 22:17:56.314464 10783 ts_manager.cc:194] Registered new tserver with Master: 4c8e47316ba24641b751aa0ccbe6ba78 (127.1.129.65:33501)
I20260923 22:17:56.314906 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 10843
I20260923 22:17:56.315016 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:56.315181 1541 external_mini_cluster.cc:1428] Setting key 73f5afba9e4f4579d2592130336ae80e
I20260923 22:17:56.315181 10783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.65:46283
I20260923 22:17:56.316040 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34777
--builtin_ntp_servers=127.1.129.84:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 22:17:56.395726 10974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:56.395920 10974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:56.395942 10974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:56.397105 10974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:56.397157 10974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:56.397184 10974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:56.398521 10974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:56.398782 10974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:56.399096 10974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:56.403923 10980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.403956 10982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.403954 10979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.404134 10974 server_base.cc:1034] running on GCE node
I20260923 22:17:56.404368 10974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:56.404592 10974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:56.405743 10974 hybrid_clock.cc:648] HybridClock initialized: now 1790201876405734 us; error 29 us; skew 500 ppm
I20260923 22:17:56.407040 10974 webserver.cc:460] Webserver started at http://127.1.129.66:33227/ using document root <none> and password file <none>
I20260923 22:17:56.407264 10974 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:56.407307 10974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:56.407469 10974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:56.408471 10974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "be4574ecc8f14bc1b1b23e1ae41ab493"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "26d9a6518dabfb35a31b57eab3aa7d29"
server_key_iv: "6033d0c88edc5cdface8f2ee0b1936f7"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.408804 10974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "be4574ecc8f14bc1b1b23e1ae41ab493"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "26d9a6518dabfb35a31b57eab3aa7d29"
server_key_iv: "6033d0c88edc5cdface8f2ee0b1936f7"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.410090 10974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:56.410832 10988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.411002 10974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.411067 10974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "be4574ecc8f14bc1b1b23e1ae41ab493"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "26d9a6518dabfb35a31b57eab3aa7d29"
server_key_iv: "6033d0c88edc5cdface8f2ee0b1936f7"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.411116 10974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:56.419875 10974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:56.420107 10974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:56.420207 10974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:56.420429 10974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:56.420703 10974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:56.420738 10974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.420759 10974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:56.420773 10974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.426287 10974 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:36149
I20260923 22:17:56.426319 11101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:36149 every 8 connection(s)
I20260923 22:17:56.426694 10974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:56.430614 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 10974
I20260923 22:17:56.430740 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:56.430934 1541 external_mini_cluster.cc:1428] Setting key 0cf38c7ba781d11f89317dc099805703
I20260923 22:17:56.431699 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.427433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:51967 (local address 127.1.129.126:34777)
0923 22:17:56.427476 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.427478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.427717 (+ 239us) server_negotiation.cc:407] Connection header received
0923 22:17:56.428175 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.428176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.428189 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.428241 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.428850 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.429316 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.430238 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.430439 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.430508 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.431415 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:56.431417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:56.431417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:56.431422 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:56.431437 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:56.431441 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:56.431496 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:56.431618 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 22:17:56.431643 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:17:56.431833 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34777
--builtin_ntp_servers=127.1.129.84:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 22:17:56.431969 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.427453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34777 (local address 127.1.129.66:51967)
0923 22:17:56.427596 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:56.427627 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:56.428096 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:56.428293 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:56.428296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:56.428304 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:56.428785 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.428788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.429389 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.429389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.430170 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.430172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.430516 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.430517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.430618 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.431265 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:56.431265 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:56.431309 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:56.431331 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:56.431517 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:56.431537 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:56.431541 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:56.431618 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 22:17:56.431726 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260923 22:17:56.433326 11102 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34777
I20260923 22:17:56.433414 11102 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:56.433597 11102 heartbeater.cc:511] Master 127.1.129.126:34777 requested a full tablet report, sending...
I20260923 22:17:56.433939 10783 ts_manager.cc:194] Registered new tserver with Master: be4574ecc8f14bc1b1b23e1ae41ab493 (127.1.129.66:36149)
I20260923 22:17:56.434295 10783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.66:51967
W20260923 22:17:56.511037 11105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:56.511262 11105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:56.511297 11105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:56.512400 11105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:56.512465 11105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:56.512506 11105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:56.513893 11105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:56.514149 11105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:56.514453 11105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:56.519462 11111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.519479 11113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.519428 11110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.519608 11105 server_base.cc:1034] running on GCE node
I20260923 22:17:56.519822 11105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:56.520092 11105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:56.521255 11105 hybrid_clock.cc:648] HybridClock initialized: now 1790201876521249 us; error 34 us; skew 500 ppm
I20260923 22:17:56.522516 11105 webserver.cc:460] Webserver started at http://127.1.129.67:45421/ using document root <none> and password file <none>
I20260923 22:17:56.522809 11105 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:56.522888 11105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:56.523025 11105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:56.524047 11105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "fcdcb4cde057413da3d7796852a0f4e2"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "c93a9f18597052c4592cef8691c71190"
server_key_iv: "e2d38c70f604f76635ae53804f135a03"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.524408 11105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "fcdcb4cde057413da3d7796852a0f4e2"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "c93a9f18597052c4592cef8691c71190"
server_key_iv: "e2d38c70f604f76635ae53804f135a03"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.525661 11105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:56.526484 11119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.526678 11105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:56.526778 11105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "fcdcb4cde057413da3d7796852a0f4e2"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "c93a9f18597052c4592cef8691c71190"
server_key_iv: "e2d38c70f604f76635ae53804f135a03"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.526875 11105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:56.552663 11105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:56.552989 11105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:56.553143 11105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:56.553362 11105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:56.553692 11105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:56.553745 11105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.553785 11105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:56.553813 11105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.559512 11105 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:40863
I20260923 22:17:56.559587 11232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:40863 every 8 connection(s)
I20260923 22:17:56.559930 11105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:56.564910 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.560715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:42303 (local address 127.1.129.126:34777)
0923 22:17:56.560792 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.560794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.561019 (+ 225us) server_negotiation.cc:407] Connection header received
0923 22:17:56.561471 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.561472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.561482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.561541 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.562168 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.562720 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.563652 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.563839 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.563888 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.564713 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:56.564714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:56.564714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:56.564718 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:56.564729 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:56.564732 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:56.564768 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:56.564831 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 22:17:56.564853 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:56.565140 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.560706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34777 (local address 127.1.129.67:42303)
0923 22:17:56.560884 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:56.560926 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:56.561388 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:56.561593 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:56.561596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:56.561604 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:56.562095 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.562099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.562800 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.562801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.563575 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.563577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.563914 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.563914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.564017 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.564590 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:56.564590 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:56.564617 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:56.564640 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:56.564769 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:56.564771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:56.564773 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:56.564819 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 22:17:56.564896 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260923 22:17:56.565701 11233 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34777
I20260923 22:17:56.565806 11233 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:56.565996 11233 heartbeater.cc:511] Master 127.1.129.126:34777 requested a full tablet report, sending...
I20260923 22:17:56.566358 10783 ts_manager.cc:194] Registered new tserver with Master: fcdcb4cde057413da3d7796852a0f4e2 (127.1.129.67:40863)
I20260923 22:17:56.566808 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 11105
I20260923 22:17:56.566826 10783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.67:42303
I20260923 22:17:56.566891 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:56.567082 1541 external_mini_cluster.cc:1428] Setting key e310b532735a78ee7306c5acbbed3bba
I20260923 22:17:56.567837 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 22:17:56.569551 10823 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:56.569096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.1.129.126:34777)
0923 22:17:56.569143 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.569145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.569261 (+ 116us) server_negotiation.cc:407] Connection header received
0923 22:17:56.569300 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.569300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.569311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.569355 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.569500 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44604: BlockingRecv error: recv got EOF from 127.0.0.1:44604 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:56.569602 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 10843
I20260923 22:17:56.574147 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 10974
I20260923 22:17:56.578488 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 11105
I20260923 22:17:56.582602 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 10752
2026-09-23T22:17:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (601 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T22:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:56Z Disabled control of system clock
I20260923 22:17:56.608846 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44615
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 22:17:56.686980 11244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:56.687177 11244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:56.687198 11244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:56.688263 11244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:56.688298 11244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:56.688311 11244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:56.688324 11244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:56.688336 11244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:56.689584 11244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:44615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:56.689769 11244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:56.690030 11244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:56.694860 11244 server_base.cc:1034] running on GCE node
W20260923 22:17:56.694844 11249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.694830 11250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.694844 11252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.695312 11244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:56.695641 11244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:56.696776 11244 hybrid_clock.cc:648] HybridClock initialized: now 1790201876696759 us; error 38 us; skew 500 ppm
I20260923 22:17:56.698000 11244 webserver.cc:460] Webserver started at http://127.1.129.126:32831/ using document root <none> and password file <none>
I20260923 22:17:56.698248 11244 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:56.698321 11244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:56.698490 11244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:56.699599 11244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "c7f6ace6c9a7417db7220d1006bcbd46"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "6388d06b069fbe2c25f991c918a7347e"
server_key_iv: "1b54269fd2c21e360e367436573cfd02"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.699955 11244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "c7f6ace6c9a7417db7220d1006bcbd46"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "6388d06b069fbe2c25f991c918a7347e"
server_key_iv: "1b54269fd2c21e360e367436573cfd02"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.701165 11244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:56.701849 11258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.702021 11244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:56.702118 11244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "c7f6ace6c9a7417db7220d1006bcbd46"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "6388d06b069fbe2c25f991c918a7347e"
server_key_iv: "1b54269fd2c21e360e367436573cfd02"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.702203 11244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:56.705449 11244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:56.705698 11244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:56.705834 11244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:56.709539 11244 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:44615
I20260923 22:17:56.709641 11310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:44615 every 8 connection(s)
I20260923 22:17:56.709934 11244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:56.710568 11311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:56.712673 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 11244
I20260923 22:17:56.712769 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:56.712929 1541 external_mini_cluster.cc:1428] Setting key 49a2fa412cb594060fd3bbe3328d1e54
I20260923 22:17:56.713479 11311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46: Bootstrap starting.
I20260923 22:17:56.714167 11311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:56.714490 11311 log.cc:824] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:56.715202 11311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46: No bootstrap required, opened a new log
I20260923 22:17:56.715870 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.713529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.1.129.126:44615)
0923 22:17:56.713703 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.713712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.713743 (+ 31us) server_negotiation.cc:407] Connection header received
0923 22:17:56.714362 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.714370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.714425 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.714578 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.715386 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:56.715390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:56.715390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:56.715394 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:56.715444 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:56.715454 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:56.715515 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:56.715609 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 22:17:56.715674 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260923 22:17:56.716711 11311 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f6ace6c9a7417db7220d1006bcbd46" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44615 } }
I20260923 22:17:56.716863 11311 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:56.716912 11311 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f6ace6c9a7417db7220d1006bcbd46, State: Initialized, Role: FOLLOWER
I20260923 22:17:56.717036 11311 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f6ace6c9a7417db7220d1006bcbd46" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44615 } }
I20260923 22:17:56.717387 11311 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:56.717434 11311 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:56.717486 11311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:56.718068 11311 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f6ace6c9a7417db7220d1006bcbd46" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44615 } }
I20260923 22:17:56.718180 11311 leader_election.cc:302] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f6ace6c9a7417db7220d1006bcbd46; no voters:
I20260923 22:17:56.718353 11311 leader_election.cc:288] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:56.718408 11316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:56.718567 11316 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [term 1 LEADER]: Becoming Leader. State: Replica: c7f6ace6c9a7417db7220d1006bcbd46, State: Running, Role: LEADER
I20260923 22:17:56.718657 11311 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:56.718683 11316 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f6ace6c9a7417db7220d1006bcbd46" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44615 } }
I20260923 22:17:56.719079 11317 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f6ace6c9a7417db7220d1006bcbd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f6ace6c9a7417db7220d1006bcbd46" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44615 } } }
I20260923 22:17:56.719216 11317 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:56.719575 11324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:56.720080 11324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:56.719071 11318 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f6ace6c9a7417db7220d1006bcbd46. Latest consensus state: current_term: 1 leader_uuid: "c7f6ace6c9a7417db7220d1006bcbd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f6ace6c9a7417db7220d1006bcbd46" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44615 } } }
I20260923 22:17:56.720644 11318 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:56.721472 11324 catalog_manager.cc:1347] Generated new cluster ID: 93e6c16e0b31476bb445e031fe680eda
I20260923 22:17:56.721518 11324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:56.735348 11324 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:56.735524 11324 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:56.738915 11324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f6ace6c9a7417db7220d1006bcbd46: Generated new TSK 0
I20260923 22:17:56.739095 11324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:56.741496 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44615
--builtin_ntp_servers=127.1.129.84:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 22:17:56.821530 11335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:56.821739 11335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:56.821762 11335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:56.823028 11335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:56.823067 11335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:56.823096 11335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:56.824468 11335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:56.824713 11335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:56.825031 11335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:56.830691 11343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.830698 11340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.830876 11335 server_base.cc:1034] running on GCE node
W20260923 22:17:56.830705 11341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.831168 11335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:56.831425 11335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:56.832588 11335 hybrid_clock.cc:648] HybridClock initialized: now 1790201876832576 us; error 29 us; skew 500 ppm
I20260923 22:17:56.833847 11335 webserver.cc:460] Webserver started at http://127.1.129.65:38447/ using document root <none> and password file <none>
I20260923 22:17:56.834085 11335 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:56.834154 11335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:56.834304 11335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:56.835419 11335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "4cef6c389cee43258d7459664f541778"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "b53545331259e16e9b8a46e785224f2f"
server_key_iv: "9ccddb8c64c6a1dbd0d9749cdfa71657"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.835785 11335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "4cef6c389cee43258d7459664f541778"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "b53545331259e16e9b8a46e785224f2f"
server_key_iv: "9ccddb8c64c6a1dbd0d9749cdfa71657"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.837073 11335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:56.837836 11349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.838027 11335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:56.838115 11335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "4cef6c389cee43258d7459664f541778"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "b53545331259e16e9b8a46e785224f2f"
server_key_iv: "9ccddb8c64c6a1dbd0d9749cdfa71657"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.838202 11335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:56.841452 11335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:56.841708 11335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:56.841858 11335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:56.842072 11335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:56.842375 11335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:56.842434 11335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.842471 11335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:56.842497 11335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.847980 11335 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:45805
I20260923 22:17:56.848047 11462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:45805 every 8 connection(s)
I20260923 22:17:56.848354 11335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:56.851199 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.849015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:58269 (local address 127.1.129.126:44615)
0923 22:17:56.849092 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.849095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.849329 (+ 234us) server_negotiation.cc:407] Connection header received
0923 22:17:56.849746 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.849747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.849751 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.849807 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.850928 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:56.850930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:56.850930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:56.850935 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:56.850949 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:56.850953 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:56.851017 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:56.851134 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 22:17:56.851148 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:17:56.851297 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.848990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44615 (local address 127.1.129.65:58269)
0923 22:17:56.849197 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:56.849229 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:56.849669 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:56.849857 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:56.849860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:56.849868 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:56.850768 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:56.850769 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:56.850796 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:56.850824 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:56.851017 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:56.851019 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:56.851021 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:56.851073 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:17:56.851120 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260923 22:17:56.851780 11463 heartbeater.cc:347] Connected to a master server at 127.1.129.126:44615
I20260923 22:17:56.851856 11463 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:56.852028 11463 heartbeater.cc:511] Master 127.1.129.126:44615 requested a full tablet report, sending...
I20260923 22:17:56.852530 11275 ts_manager.cc:194] Registered new tserver with Master: 4cef6c389cee43258d7459664f541778 (127.1.129.65:45805)
I20260923 22:17:56.856092 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 11335
I20260923 22:17:56.856164 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:56.856376 1541 external_mini_cluster.cc:1428] Setting key 9f1f6f193873cb44b1a06ccdaf086505
I20260923 22:17:56.857275 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44615
--builtin_ntp_servers=127.1.129.84:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 22:17:56.936074 11466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:56.936295 11466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:56.936318 11466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:56.937438 11466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:56.937471 11466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:56.937495 11466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:56.938769 11466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:56.938947 11466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:56.939180 11466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:56.944355 11472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.944380 11474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:56.944355 11471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:56.944773 11466 server_base.cc:1034] running on GCE node
I20260923 22:17:56.944995 11466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:56.945282 11466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:56.946446 11466 hybrid_clock.cc:648] HybridClock initialized: now 1790201876946428 us; error 32 us; skew 500 ppm
I20260923 22:17:56.947741 11466 webserver.cc:460] Webserver started at http://127.1.129.66:41643/ using document root <none> and password file <none>
I20260923 22:17:56.947932 11466 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:56.947973 11466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:56.948060 11466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:56.949050 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "174a06c645ab476893681332a0c2b68b"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "016d9bcd42deb15b01d0ee481e719031"
server_key_iv: "dcd6ee849d9f912e3fcd5d82831bbc9a"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.949337 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "174a06c645ab476893681332a0c2b68b"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "016d9bcd42deb15b01d0ee481e719031"
server_key_iv: "dcd6ee849d9f912e3fcd5d82831bbc9a"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.950599 11466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:56.951551 11480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.951716 11466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:56.951802 11466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "174a06c645ab476893681332a0c2b68b"
format_stamp: "Formatted at 2026-09-23 22:17:56 on dist-test-slave-dmw7"
server_key: "016d9bcd42deb15b01d0ee481e719031"
server_key_iv: "dcd6ee849d9f912e3fcd5d82831bbc9a"
server_key_version: "encryptionkey@0"
I20260923 22:17:56.951871 11466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:56.955120 11466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:56.955374 11466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:56.955507 11466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:56.955719 11466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:56.955996 11466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:56.956044 11466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.956084 11466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:56.956110 11466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:56.961526 11466 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:35855
I20260923 22:17:56.961561 11593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:35855 every 8 connection(s)
I20260923 22:17:56.961869 11466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:56.964857 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.962679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:38495 (local address 127.1.129.126:44615)
0923 22:17:56.962736 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.962738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.962970 (+ 232us) server_negotiation.cc:407] Connection header received
0923 22:17:56.963452 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.963453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.963460 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.963518 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.964602 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:56.964604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:56.964605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:56.964609 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:56.964625 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:56.964629 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:56.964676 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:56.964772 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 22:17:56.964787 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:17:56.965066 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.962644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44615 (local address 127.1.129.66:38495)
0923 22:17:56.962856 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:56.962886 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:56.963352 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:56.963595 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:56.963598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:56.963606 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:56.964455 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:56.964455 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:56.964481 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:56.964508 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:56.964685 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:56.964689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:56.964693 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:56.964794 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 22:17:56.964854 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260923 22:17:56.965605 11594 heartbeater.cc:347] Connected to a master server at 127.1.129.126:44615
I20260923 22:17:56.965731 11594 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:56.965884 11594 heartbeater.cc:511] Master 127.1.129.126:44615 requested a full tablet report, sending...
I20260923 22:17:56.966223 11275 ts_manager.cc:194] Registered new tserver with Master: 174a06c645ab476893681332a0c2b68b (127.1.129.66:35855)
I20260923 22:17:56.971657 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 11466
I20260923 22:17:56.971766 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:56.971948 1541 external_mini_cluster.cc:1428] Setting key 2b47b1e768f49b712bfac462345bba1b
I20260923 22:17:56.972911 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44615
--builtin_ntp_servers=127.1.129.84:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 22:17:57.051975 11597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:57.052165 11597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:57.052186 11597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:57.053308 11597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:57.053345 11597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:57.053398 11597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:57.054764 11597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:57.054996 11597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:57.055292 11597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:57.060108 11597 server_base.cc:1034] running on GCE node
W20260923 22:17:57.060061 11602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:57.060082 11605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:57.060062 11603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:57.060418 11597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:57.060632 11597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:57.061779 11597 hybrid_clock.cc:648] HybridClock initialized: now 1790201877061760 us; error 36 us; skew 500 ppm
I20260923 22:17:57.063027 11597 webserver.cc:460] Webserver started at http://127.1.129.67:44529/ using document root <none> and password file <none>
I20260923 22:17:57.063267 11597 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:57.063314 11597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:57.063458 11597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:57.064455 11597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "63b5c24f36184f609d0ba0ecb170cd33"
format_stamp: "Formatted at 2026-09-23 22:17:57 on dist-test-slave-dmw7"
server_key: "c093bd06317da181422b6e4557ea96f6"
server_key_iv: "2cbe372aac590c5cb569a91b408c016a"
server_key_version: "encryptionkey@0"
I20260923 22:17:57.064792 11597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "63b5c24f36184f609d0ba0ecb170cd33"
format_stamp: "Formatted at 2026-09-23 22:17:57 on dist-test-slave-dmw7"
server_key: "c093bd06317da181422b6e4557ea96f6"
server_key_iv: "2cbe372aac590c5cb569a91b408c016a"
server_key_version: "encryptionkey@0"
I20260923 22:17:57.065999 11597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:57.066691 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:57.066875 11597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:57.066968 11597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "63b5c24f36184f609d0ba0ecb170cd33"
format_stamp: "Formatted at 2026-09-23 22:17:57 on dist-test-slave-dmw7"
server_key: "c093bd06317da181422b6e4557ea96f6"
server_key_iv: "2cbe372aac590c5cb569a91b408c016a"
server_key_version: "encryptionkey@0"
I20260923 22:17:57.067046 11597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:57.070168 11597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:57.070412 11597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:57.070545 11597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:57.070766 11597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:57.071058 11597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:57.071110 11597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:57.071148 11597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:57.071177 11597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:57.076848 11597 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:39009
I20260923 22:17:57.076880 11724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:39009 every 8 connection(s)
I20260923 22:17:57.077193 11597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:57.077373 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 11597
I20260923 22:17:57.077469 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:57.077672 1541 external_mini_cluster.cc:1428] Setting key eab9972c1b578bab6801446f7dc0bcdc
I20260923 22:17:57.080257 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:57.077960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:40575 (local address 127.1.129.126:44615)
0923 22:17:57.078012 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:57.078015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:57.078306 (+ 291us) server_negotiation.cc:407] Connection header received
0923 22:17:57.078905 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:57.078906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:57.078912 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:57.078956 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:57.080028 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:57.080030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:57.080030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:57.080035 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:57.080050 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:57.080054 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:57.080101 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:57.080176 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 22:17:57.080190 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:57.080462 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:57.077957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44615 (local address 127.1.129.67:40575)
0923 22:17:57.078200 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:57.078243 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:57.078804 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:57.079023 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:57.079028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:57.079037 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:57.079880 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:57.079880 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:57.079907 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:57.079933 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:57.080098 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:57.080103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:57.080106 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:57.080183 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 22:17:57.080242 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260923 22:17:57.080904 11725 heartbeater.cc:347] Connected to a master server at 127.1.129.126:44615
I20260923 22:17:57.080971 11725 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:57.081125 11725 heartbeater.cc:511] Master 127.1.129.126:44615 requested a full tablet report, sending...
I20260923 22:17:57.081470 11275 ts_manager.cc:194] Registered new tserver with Master: 63b5c24f36184f609d0ba0ecb170cd33 (127.1.129.67:39009)
I20260923 22:17:57.089572 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 22:17:57.091444 11314 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:57.090932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.1.129.126:44615)
0923 22:17:57.090996 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:57.090998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:57.091147 (+ 149us) server_negotiation.cc:407] Connection header received
0923 22:17:57.091185 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:57.091185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:57.091190 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:57.091235 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:57.091363 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57842: BlockingRecv error: recv got EOF from 127.0.0.1:57842 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:57.091475 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 11335
I20260923 22:17:57.095865 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 11466
I20260923 22:17:57.100128 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 11597
I20260923 22:17:57.104076 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 11244
2026-09-23T22:17:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (521 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:57 dist-test-slave-dmw7 krb5kdc[11735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:57 dist-test-slave-dmw7 krb5kdc[11735](info): set up 2 sockets
Sep 23 22:17:57 dist-test-slave-dmw7 krb5kdc[11735](info): commencing operation
krb5kdc: starting...
W20260923 22:17:58.143496 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:17:58.258277 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:42753
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:42753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:58.336388 11748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:58.336596 11748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:58.336616 11748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:58.337687 11748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:58.337720 11748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:58.337734 11748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:58.337746 11748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:58.337759 11748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:58.339205 11748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:42753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:42753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:58.339398 11748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:58.339565 11748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:58.344468 11754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:58.344460 11753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:58.344463 11756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:58.344779 11748 server_base.cc:1034] running on GCE node
I20260923 22:17:58.345021 11748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:58.345265 11748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:58.346397 11748 hybrid_clock.cc:648] HybridClock initialized: now 1790201878346388 us; error 31 us; skew 500 ppm
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:58.348901 11748 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:17:58.349334 11748 webserver.cc:460] Webserver started at http://127.1.129.126:33865/ using document root <none> and password file <none>
I20260923 22:17:58.349571 11748 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:58.349617 11748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:58.349771 11748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:58.350916 11748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "5f64d394cc1d454eb436d81e5bbe85e6"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "4b80af38ee1a2051110cc30e24f7f6d6"
server_key_iv: "6c0015bebf0beb05a2870d4b5ca7969a"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.351259 11748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "5f64d394cc1d454eb436d81e5bbe85e6"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "4b80af38ee1a2051110cc30e24f7f6d6"
server_key_iv: "6c0015bebf0beb05a2870d4b5ca7969a"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.352595 11748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 22:17:58.353303 11763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.353472 11748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.353562 11748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "5f64d394cc1d454eb436d81e5bbe85e6"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "4b80af38ee1a2051110cc30e24f7f6d6"
server_key_iv: "6c0015bebf0beb05a2870d4b5ca7969a"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.353637 11748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:58.375285 11748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:58.377244 11748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:58.377408 11748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:58.381204 11748 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:42753
I20260923 22:17:58.381243 11815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:42753 every 8 connection(s)
I20260923 22:17:58.381603 11748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:58.382562 11816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:58.382629 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 11748
I20260923 22:17:58.382755 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:17:58.382939 1541 external_mini_cluster.cc:1428] Setting key 61aa8512c4300a7b3b26e9240edddcfc
I20260923 22:17:58.385941 11816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6: Bootstrap starting.
I20260923 22:17:58.386608 11816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:58.386955 11816 log.cc:824] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:58.387676 11816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6: No bootstrap required, opened a new log
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:58.389690 11816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d394cc1d454eb436d81e5bbe85e6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 42753 } }
I20260923 22:17:58.389861 11816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:58.389920 11816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f64d394cc1d454eb436d81e5bbe85e6, State: Initialized, Role: FOLLOWER
I20260923 22:17:58.390058 11816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d394cc1d454eb436d81e5bbe85e6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 42753 } }
I20260923 22:17:58.390162 11816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:58.390204 11816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:58.390259 11816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:58.391016 11816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d394cc1d454eb436d81e5bbe85e6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 42753 } }
I20260923 22:17:58.391150 11816 leader_election.cc:302] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f64d394cc1d454eb436d81e5bbe85e6; no voters:
I20260923 22:17:58.391314 11816 leader_election.cc:288] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:58.391350 11821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:58.391486 11821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f64d394cc1d454eb436d81e5bbe85e6, State: Running, Role: LEADER
I20260923 22:17:58.391575 11816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:58.391564 11821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d394cc1d454eb436d81e5bbe85e6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 42753 } }
I20260923 22:17:58.391971 11823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f64d394cc1d454eb436d81e5bbe85e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d394cc1d454eb436d81e5bbe85e6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 42753 } } }
I20260923 22:17:58.392076 11823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:58.391956 11821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f64d394cc1d454eb436d81e5bbe85e6. Latest consensus state: current_term: 1 leader_uuid: "5f64d394cc1d454eb436d81e5bbe85e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d394cc1d454eb436d81e5bbe85e6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 42753 } } }
I20260923 22:17:58.392271 11821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:58.392406 11829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:58.393132 11829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:58.394016 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.383468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.1.129.126:42753)
0923 22:17:58.383632 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.383641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.383672 (+ 31us) server_negotiation.cc:407] Connection header received
0923 22:17:58.384172 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.384180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.384244 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.384402 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:58.384992 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.386350 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.387177 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.387714 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.387835 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.390023 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:58.390028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:58.390029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:58.390045 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:58.391997 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.392398 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.392401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.392402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.392529 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.392751 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.392752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.392752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.393083 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:58.393228 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:58.393704 (+ 476us) server_negotiation.cc:299] Negotiation successful
0923 22:17:58.393858 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260923 22:17:58.394822 11829 catalog_manager.cc:1347] Generated new cluster ID: 660ecb92cfb24f86b602accb2ea59500
I20260923 22:17:58.394874 11829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:58.399400 11829 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:58.399895 11829 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:58.406327 11829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6: Generated new TSK 0
I20260923 22:17:58.406512 11829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:17:58.439628 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:42753
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:58.519384 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:58.519608 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:58.519644 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:58.520745 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:58.520803 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:58.520845 11842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:17:58.522316 11842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:42753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:58.522560 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:58.522800 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:58.527547 11850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:58.527547 11847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:58.527618 11848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:58.527735 11842 server_base.cc:1034] running on GCE node
I20260923 22:17:58.527921 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:58.528162 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:58.529325 11842 hybrid_clock.cc:648] HybridClock initialized: now 1790201878529310 us; error 31 us; skew 500 ppm
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:58.531822 11842 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:17:58.532275 11842 webserver.cc:460] Webserver started at http://127.1.129.65:41909/ using document root <none> and password file <none>
I20260923 22:17:58.532500 11842 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:58.532563 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:58.532704 11842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:58.533715 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "4c2688bd70e3411c85db0a588adde80a"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "c3e54d105488afa4fac72b5e6b7d4f1f"
server_key_iv: "e768a08b79aacf86ce91456afe0c8ed1"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.534061 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "4c2688bd70e3411c85db0a588adde80a"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "c3e54d105488afa4fac72b5e6b7d4f1f"
server_key_iv: "e768a08b79aacf86ce91456afe0c8ed1"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.535382 11842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:58.536042 11857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.536197 11842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.536291 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "4c2688bd70e3411c85db0a588adde80a"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "c3e54d105488afa4fac72b5e6b7d4f1f"
server_key_iv: "e768a08b79aacf86ce91456afe0c8ed1"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.536360 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:58.558868 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:58.560853 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:58.561022 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:58.561259 11842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:58.561568 11842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:58.561617 11842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.561658 11842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:58.561686 11842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.567478 11842 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:38255
I20260923 22:17:58.567530 11970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:38255 every 8 connection(s)
I20260923 22:17:58.567831 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:58.574942 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 11842
I20260923 22:17:58.575016 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:58.575186 1541 external_mini_cluster.cc:1428] Setting key e9cf673a7ea2858ed0ed017441576535
I20260923 22:17:58.576807 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.568554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:53271 (local address 127.1.129.126:42753)
0923 22:17:58.568618 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.568620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.568838 (+ 218us) server_negotiation.cc:407] Connection header received
0923 22:17:58.569311 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.569312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.569322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.569364 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:58.570264 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.570771 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.571676 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.571875 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.571923 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.573982 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:58.573983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:58.573983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:58.573989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:58.575403 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.575844 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.575846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.575846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.575932 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.576182 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.576183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.576183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.576298 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:58.576369 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:58.576638 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 22:17:58.576720 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:17:58.577222 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.568533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:42753 (local address 127.1.129.65:53271)
0923 22:17:58.568723 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.568760 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.569223 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.569417 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.569420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.570001 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:58.570190 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.570194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.570846 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.570847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.571613 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.571615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.571941 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.571942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.572045 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.572595 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:58.572602 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:58.573904 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:58.575491 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:58.575497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:58.575499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:58.575778 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:58.576005 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:58.576006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:58.576006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:58.576117 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:58.576370 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:58.576377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:58.576507 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 22:17:58.576605 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 22:17:58.576761 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260923 22:17:58.577693 11971 heartbeater.cc:347] Connected to a master server at 127.1.129.126:42753
I20260923 22:17:58.577817 11971 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:58.577999 11971 heartbeater.cc:511] Master 127.1.129.126:42753 requested a full tablet report, sending...
I20260923 22:17:58.578454 11780 ts_manager.cc:194] Registered new tserver with Master: 4c2688bd70e3411c85db0a588adde80a (127.1.129.65:38255)
I20260923 22:17:58.579110 11780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:53271
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:17:58.601207 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:42753
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:58.681708 11976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:58.681957 11976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:58.681996 11976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:58.683218 11976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:58.683290 11976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:58.683337 11976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:17:58.684788 11976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:42753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:58.685051 11976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:58.685288 11976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:58.690431 11984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:58.690558 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:58.690606 11976 server_base.cc:1034] running on GCE node
W20260923 22:17:58.690691 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:58.690907 11976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:58.691150 11976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:58.692319 11976 hybrid_clock.cc:648] HybridClock initialized: now 1790201878692302 us; error 35 us; skew 500 ppm
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:58.694828 11976 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:17:58.695240 11976 webserver.cc:460] Webserver started at http://127.1.129.66:41597/ using document root <none> and password file <none>
I20260923 22:17:58.695478 11976 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:58.695565 11976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:58.695698 11976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:58.696727 11976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "ca5b5c790e697db6f2e55f02c9712cb1"
server_key_iv: "1e341a63a223104be5dc50d8e4d1e845"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.697060 11976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "ca5b5c790e697db6f2e55f02c9712cb1"
server_key_iv: "1e341a63a223104be5dc50d8e4d1e845"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.698319 11976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:58.699069 11991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.699250 11976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.699330 11976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "ca5b5c790e697db6f2e55f02c9712cb1"
server_key_iv: "1e341a63a223104be5dc50d8e4d1e845"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.699419 11976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:58.715502 11976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:58.717413 11976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:58.717550 11976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:58.717782 11976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:58.718081 11976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:58.718113 11976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.718151 11976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:58.718178 11976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.723758 11976 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:34717
I20260923 22:17:58.723798 12104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:34717 every 8 connection(s)
I20260923 22:17:58.724118 11976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:58.725991 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 11976
I20260923 22:17:58.726066 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:58.726248 1541 external_mini_cluster.cc:1428] Setting key e07176532443579cd8cf7528e35b069b
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:58.733372 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.724827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:45321 (local address 127.1.129.126:42753)
0923 22:17:58.724869 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.724871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.725059 (+ 188us) server_negotiation.cc:407] Connection header received
0923 22:17:58.725545 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.725546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.725555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.725607 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:58.726706 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.727398 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.728422 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.728625 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.728705 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.730744 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:58.730745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:58.730745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:58.730752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:58.732076 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.732474 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.732475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.732476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.732534 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.732756 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.732757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.732758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.732875 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:58.732947 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:58.733198 (+ 251us) server_negotiation.cc:299] Negotiation successful
0923 22:17:58.733280 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:17:58.733558 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.724806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:42753 (local address 127.1.129.66:45321)
0923 22:17:58.724940 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.724973 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.725466 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.725658 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.725662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.726305 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:58.726588 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.726594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.727496 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.727497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.728334 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.728337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.728713 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.728714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.728848 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.729332 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:58.729340 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:58.730614 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:58.732155 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:58.732161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:58.732164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:58.732388 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:58.732587 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:58.732587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:58.732588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:58.732691 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:58.732938 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:58.732944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:58.733076 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 22:17:58.733177 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 22:17:58.733339 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
I20260923 22:17:58.734011 12105 heartbeater.cc:347] Connected to a master server at 127.1.129.126:42753
I20260923 22:17:58.734091 12105 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:58.734272 12105 heartbeater.cc:511] Master 127.1.129.126:42753 requested a full tablet report, sending...
I20260923 22:17:58.734625 11780 ts_manager.cc:194] Registered new tserver with Master: 3c35e4ffb5d242dc91cd6ff0a22bfc55 (127.1.129.66:34717)
I20260923 22:17:58.735075 11780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:45321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:17:58.749094 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:42753
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:58.828311 12110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:58.828526 12110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:58.828549 12110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:58.829646 12110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:58.829684 12110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:58.829708 12110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:17:58.831174 12110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:42753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:17:58.831363 12110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:58.831539 12110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:58.836522 12116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:58.836581 12115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:58.836522 12118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:58.836576 12110 server_base.cc:1034] running on GCE node
I20260923 22:17:58.836911 12110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:58.837173 12110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:58.838320 12110 hybrid_clock.cc:648] HybridClock initialized: now 1790201878838314 us; error 40 us; skew 500 ppm
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:58.840999 12110 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:17:58.841459 12110 webserver.cc:460] Webserver started at http://127.1.129.67:32899/ using document root <none> and password file <none>
I20260923 22:17:58.841673 12110 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:58.841735 12110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:58.841878 12110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:58.843011 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "0174542109504565b898bcd6b89dfa71"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "19089ee7bfed30175351c34b68787ca4"
server_key_iv: "b1d69c975500f63ece80efba8e51f1b6"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.843348 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "0174542109504565b898bcd6b89dfa71"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "19089ee7bfed30175351c34b68787ca4"
server_key_iv: "b1d69c975500f63ece80efba8e51f1b6"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.844587 12110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:58.845253 12125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.845497 12110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:58.845590 12110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "0174542109504565b898bcd6b89dfa71"
format_stamp: "Formatted at 2026-09-23 22:17:58 on dist-test-slave-dmw7"
server_key: "19089ee7bfed30175351c34b68787ca4"
server_key_iv: "b1d69c975500f63ece80efba8e51f1b6"
server_key_version: "encryptionkey@0"
I20260923 22:17:58.845665 12110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:58.857843 12110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:58.860141 12110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:58.860291 12110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:58.860513 12110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:58.860813 12110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:58.860863 12110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.860913 12110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:58.860937 12110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:58.866456 12110 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:46193
I20260923 22:17:58.866499 12238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:46193 every 8 connection(s)
I20260923 22:17:58.866888 12110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:17:58 dist-test-slave-dmw7 krb5kdc[11735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201878, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:17:58.873382 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 12110
I20260923 22:17:58.873481 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:58.873704 1541 external_mini_cluster.cc:1428] Setting key 3322b4cd95c71a3d797be9614252568e
I20260923 22:17:58.875911 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.867563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:35867 (local address 127.1.129.126:42753)
0923 22:17:58.867637 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.867640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.867799 (+ 159us) server_negotiation.cc:407] Connection header received
0923 22:17:58.868268 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.868268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.868278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.868329 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:58.869224 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.869707 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.870683 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.870949 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.870999 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.872969 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:58.872970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:58.872970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:58.872976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:58.874576 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.874993 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.874994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.874995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.875047 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.875289 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.875290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.875291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.875405 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:58.875476 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:58.875740 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 22:17:58.875814 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:17:58.876096 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.867525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:42753 (local address 127.1.129.67:35867)
0923 22:17:58.867690 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.867726 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.868190 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.868384 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.868387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.868961 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:58.869151 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.869155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.869785 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.869786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.870565 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.870566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.871014 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.871015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.871119 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.871548 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:58.871555 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:58.872885 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:58.874669 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:58.874674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:58.874677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:58.874898 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:58.875117 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:58.875117 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:58.875118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:58.875225 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:58.875468 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:58.875473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:58.875609 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 22:17:58.875708 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 22:17:58.875859 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260923 22:17:58.876452 12239 heartbeater.cc:347] Connected to a master server at 127.1.129.126:42753
I20260923 22:17:58.876534 12239 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:58.876719 12239 heartbeater.cc:511] Master 127.1.129.126:42753 requested a full tablet report, sending...
I20260923 22:17:58.877094 11780 ts_manager.cc:194] Registered new tserver with Master: 0174542109504565b898bcd6b89dfa71 (127.1.129.67:46193)
I20260923 22:17:58.877435 11780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:35867
I20260923 22:17:58.885540 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:58.893211 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.886891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.1.129.126:42753)
0923 22:17:58.886942 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.886944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.887072 (+ 128us) server_negotiation.cc:407] Connection header received
0923 22:17:58.887151 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.887152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.887162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.887215 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:58.887866 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.888352 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.889083 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.889276 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.889355 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.890538 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:58.890540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:58.890540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:58.890547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:58.891888 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.892282 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.892283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.892283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.892344 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:58.892539 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:58.892540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:58.892540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:58.892689 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:58.892797 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:58.892992 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 22:17:58.893077 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:17:58.894825 11780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:58.896353 11780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:58.902349 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.898436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:46193 (local address 127.0.0.1:60158)
0923 22:17:58.898517 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.898526 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.898586 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.898952 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.898953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.898955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:58.899016 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.899018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.899930 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.899931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.901169 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.901172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.902026 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.902026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.902112 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.902126 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 22:17:58.902191 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 22:17:58.902254 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":58,"threads_started":1}
I20260923 22:17:58.902451 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.898310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.1.129.65:38255)
0923 22:17:58.898457 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.898462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.898582 (+ 120us) server_negotiation.cc:407] Connection header received
0923 22:17:58.898743 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.898746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.898763 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.898846 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:58.899036 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.900192 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.901403 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.902127 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.902225 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.902278 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:58.902334 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 22:17:58.902377 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260923 22:17:58.902781 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.898339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.1.129.66:34717)
0923 22:17:58.898849 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.898854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.898864 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:58.898906 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.898908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.898924 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.899107 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:58.899312 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.900339 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.901568 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.902352 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.902410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.902454 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:58.902666 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 22:17:58.902711 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":94,"threads_started":1}
I20260923 22:17:58.902941 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.898620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.1.129.67:46193)
0923 22:17:58.898819 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.898823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.898832 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:58.898869 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.898871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.898883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.898954 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:58.899097 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.899834 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.901258 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.901941 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.902046 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.902113 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:58.902775 (+ 662us) server_negotiation.cc:299] Negotiation successful
0923 22:17:58.902871 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260923 22:17:58.902455 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.898208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:38255 (local address 127.0.0.1:55716)
0923 22:17:58.898474 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.898516 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.898608 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.898868 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.898871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.898874 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:58.898943 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.898945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.900293 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.900294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.901328 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.901329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.902202 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.902203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.902264 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.902267 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:58.902319 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:17:58.902350 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":85,"threads_started":1}
I20260923 22:17:58.902765 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.898236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34717 (local address 127.0.0.1:45518)
0923 22:17:58.898522 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.898534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.898589 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.899184 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.899185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.899187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:58.899237 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.899238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.900435 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.900436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.901472 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.901474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.902487 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.902488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.902566 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.902570 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:58.902635 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 22:17:58.902688 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":58,"threads_started":1}
I20260923 22:17:58.903797 11905 tablet_service.cc:1467] Processing CreateTablet for tablet e1ecdb554e71426ca0351620252ee9d4 (DEFAULT_TABLE table=test-table [id=00444ee820ec48039edf8f788ed9f332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:58.903880 12173 tablet_service.cc:1467] Processing CreateTablet for tablet e1ecdb554e71426ca0351620252ee9d4 (DEFAULT_TABLE table=test-table [id=00444ee820ec48039edf8f788ed9f332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:58.904124 11905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ecdb554e71426ca0351620252ee9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:58.904176 12173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ecdb554e71426ca0351620252ee9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:58.906695 12253 tablet_bootstrap.cc:492] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Bootstrap starting.
I20260923 22:17:58.907377 12253 tablet_bootstrap.cc:654] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:58.907275 12039 tablet_service.cc:1467] Processing CreateTablet for tablet e1ecdb554e71426ca0351620252ee9d4 (DEFAULT_TABLE table=test-table [id=00444ee820ec48039edf8f788ed9f332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:58.907583 12039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ecdb554e71426ca0351620252ee9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:58.907950 12253 log.cc:824] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:58.908706 12253 tablet_bootstrap.cc:492] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: No bootstrap required, opened a new log
I20260923 22:17:58.908778 12253 ts_tablet_manager.cc:1397] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:58.910775 12256 tablet_bootstrap.cc:492] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Bootstrap starting.
I20260923 22:17:58.910924 12254 tablet_bootstrap.cc:492] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: Bootstrap starting.
I20260923 22:17:58.910959 12253 raft_consensus.cc:348] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:58.911114 12253 raft_consensus.cc:374] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:58.911159 12253 raft_consensus.cc:729] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0174542109504565b898bcd6b89dfa71, State: Initialized, Role: FOLLOWER
I20260923 22:17:58.911299 12253 consensus_queue.cc:260] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:58.911451 12256 tablet_bootstrap.cc:654] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:58.911547 12254 tablet_bootstrap.cc:654] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:58.911573 12253 ts_tablet_manager.cc:1428] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:58.911698 12239 heartbeater.cc:503] Master 127.1.129.126:42753 was elected leader, sending a full tablet report...
I20260923 22:17:58.911993 12256 log.cc:824] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:58.912083 12254 log.cc:824] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:58.912729 12256 tablet_bootstrap.cc:492] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: No bootstrap required, opened a new log
I20260923 22:17:58.912807 12256 ts_tablet_manager.cc:1397] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:58.912834 12254 tablet_bootstrap.cc:492] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: No bootstrap required, opened a new log
I20260923 22:17:58.912904 12254 ts_tablet_manager.cc:1397] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:58.914281 12256 raft_consensus.cc:348] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:58.914399 12256 raft_consensus.cc:374] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:58.914420 12256 raft_consensus.cc:729] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c35e4ffb5d242dc91cd6ff0a22bfc55, State: Initialized, Role: FOLLOWER
I20260923 22:17:58.914546 12256 consensus_queue.cc:260] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:58.914494 12254 raft_consensus.cc:348] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:58.914702 12254 raft_consensus.cc:374] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:58.914750 12254 raft_consensus.cc:729] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2688bd70e3411c85db0a588adde80a, State: Initialized, Role: FOLLOWER
I20260923 22:17:58.914835 12256 ts_tablet_manager.cc:1428] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:17:58.914896 12105 heartbeater.cc:503] Master 127.1.129.126:42753 was elected leader, sending a full tablet report...
I20260923 22:17:58.914870 12254 consensus_queue.cc:260] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:58.915125 12254 ts_tablet_manager.cc:1428] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:58.915138 11971 heartbeater.cc:503] Master 127.1.129.126:42753 was elected leader, sending a full tablet report...
W20260923 22:17:58.974761 12106 tablet.cc:2369] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:59.068531 11972 tablet.cc:2369] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:59.100317 12260 raft_consensus.cc:482] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:59.100435 12260 raft_consensus.cc:504] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:59.100752 12260 leader_election.cc:288] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c2688bd70e3411c85db0a588adde80a (127.1.129.65:38255), 0174542109504565b898bcd6b89dfa71 (127.1.129.67:46193)
I20260923 22:17:59.103734 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.101038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:46193 (local address 127.1.129.66:33361)
0923 22:17:59.101203 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:59.101212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:59.101250 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:59.101426 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:59.101427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:59.101429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:59.101493 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.101494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.102105 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.102106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.102935 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.102937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.103487 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.103488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.103556 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.103564 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 22:17:59.103606 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 22:17:59.103635 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260923 22:17:59.103747 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.101150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:33361 (local address 127.1.129.67:46193)
0923 22:17:59.101315 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:59.101318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:59.101325 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:59.101348 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:59.101349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:59.101359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:59.101414 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:59.101568 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.102030 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.103013 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.103423 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.103475 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.103481 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:59.103631 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 22:17:59.103680 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260923 22:17:59.103823 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.101025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:46125 (local address 127.1.129.65:38255)
0923 22:17:59.101075 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:59.101078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:59.101090 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:59.101154 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:59.101155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:59.101181 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:59.101242 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:59.101421 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.101967 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.103061 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.103438 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.103522 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.103529 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:59.103705 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 22:17:59.103742 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 22:17:59.103837 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.100931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:38255 (local address 127.1.129.66:46125)
0923 22:17:59.100994 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:59.101005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:59.101080 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:59.101244 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:59.101245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:59.101247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:59.101321 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.101323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.102035 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.102036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.102956 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.102958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.103534 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.103535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.103620 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.103624 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:59.103685 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:17:59.103718 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 22:17:59.104182 11925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ecdb554e71426ca0351620252ee9d4" candidate_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2688bd70e3411c85db0a588adde80a" is_pre_election: true
I20260923 22:17:59.104341 11925 raft_consensus.cc:2454] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c35e4ffb5d242dc91cd6ff0a22bfc55 in term 0.
I20260923 22:17:59.104295 12193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ecdb554e71426ca0351620252ee9d4" candidate_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174542109504565b898bcd6b89dfa71" is_pre_election: true
I20260923 22:17:59.104442 12193 raft_consensus.cc:2454] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c35e4ffb5d242dc91cd6ff0a22bfc55 in term 0.
I20260923 22:17:59.104590 11995 leader_election.cc:302] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c35e4ffb5d242dc91cd6ff0a22bfc55, 4c2688bd70e3411c85db0a588adde80a; no voters:
I20260923 22:17:59.104737 12260 raft_consensus.cc:2781] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:59.104805 12260 raft_consensus.cc:482] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:59.104822 12260 raft_consensus.cc:3037] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:59.105506 12260 raft_consensus.cc:504] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:59.105633 12260 leader_election.cc:288] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [CANDIDATE]: Term 1 election: Requested vote from peers 4c2688bd70e3411c85db0a588adde80a (127.1.129.65:38255), 0174542109504565b898bcd6b89dfa71 (127.1.129.67:46193)
I20260923 22:17:59.105808 11925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ecdb554e71426ca0351620252ee9d4" candidate_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2688bd70e3411c85db0a588adde80a"
I20260923 22:17:59.105820 12193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ecdb554e71426ca0351620252ee9d4" candidate_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0174542109504565b898bcd6b89dfa71"
I20260923 22:17:59.105891 11925 raft_consensus.cc:3037] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:59.105888 12193 raft_consensus.cc:3037] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:59.106541 12193 raft_consensus.cc:2454] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c35e4ffb5d242dc91cd6ff0a22bfc55 in term 1.
I20260923 22:17:59.106683 11925 raft_consensus.cc:2454] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c35e4ffb5d242dc91cd6ff0a22bfc55 in term 1.
I20260923 22:17:59.106712 11995 leader_election.cc:302] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0174542109504565b898bcd6b89dfa71, 3c35e4ffb5d242dc91cd6ff0a22bfc55; no voters:
I20260923 22:17:59.106789 12260 raft_consensus.cc:2781] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:59.106937 12260 raft_consensus.cc:686] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 1 LEADER]: Becoming Leader. State: Replica: 3c35e4ffb5d242dc91cd6ff0a22bfc55, State: Running, Role: LEADER
I20260923 22:17:59.107025 12260 consensus_queue.cc:237] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } }
I20260923 22:17:59.107790 11780 catalog_manager.cc:5495] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c35e4ffb5d242dc91cd6ff0a22bfc55 (127.1.129.66). New cstate: current_term: 1 leader_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c35e4ffb5d242dc91cd6ff0a22bfc55" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 } health_report { overall_health: UNKNOWN } } }
W20260923 22:17:59.117399 12240 tablet.cc:2369] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:59.127007 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.124808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.1.129.66:34717)
0923 22:17:59.124861 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:59.124863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:59.124891 (+ 28us) server_negotiation.cc:407] Connection header received
0923 22:17:59.124925 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:59.124926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:59.124941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:59.124991 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:59.125104 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.125549 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.126275 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.126452 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.126498 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.126621 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:59.126779 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:59.126840 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:59.126881 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 22:17:59.126897 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:17:59.128639 12193 raft_consensus.cc:1261] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 1 FOLLOWER]: Refusing update from remote peer 3c35e4ffb5d242dc91cd6ff0a22bfc55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:59.128639 11925 raft_consensus.cc:1261] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 1 FOLLOWER]: Refusing update from remote peer 3c35e4ffb5d242dc91cd6ff0a22bfc55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:59.128968 12263 consensus_queue.cc:1035] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0174542109504565b898bcd6b89dfa71" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:59.129083 12260 consensus_queue.cc:1035] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c2688bd70e3411c85db0a588adde80a" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:59.131150 12266 mvcc.cc:205] Tried to move back new op lower bound from 7332666896908398592 to 7332666896822788096. Current Snapshot: MvccSnapshot[applied={T|T < 7332666896908398592}]
I20260923 22:17:59.132197 12267 mvcc.cc:205] Tried to move back new op lower bound from 7332666896908398592 to 7332666896822788096. Current Snapshot: MvccSnapshot[applied={T|T < 7332666896908398592 or (T in {7332666896908398592})}]
I20260923 22:17:59.133356 11780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:59.133446 11780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:59.134119 11780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f64d394cc1d454eb436d81e5bbe85e6: Sending DeleteTablet for 3 replicas of tablet e1ecdb554e71426ca0351620252ee9d4
I20260923 22:17:59.134416 12039 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ecdb554e71426ca0351620252ee9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:59 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:45518
I20260923 22:17:59.134436 11905 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ecdb554e71426ca0351620252ee9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:59 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:55716
I20260923 22:17:59.134481 12173 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ecdb554e71426ca0351620252ee9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:59 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:60158
I20260923 22:17:59.134636 12275 tablet_replica.cc:330] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: stopping tablet replica
I20260923 22:17:59.134687 12276 tablet_replica.cc:330] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: stopping tablet replica
I20260923 22:17:59.134763 12275 raft_consensus.cc:2229] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:59.134814 12276 raft_consensus.cc:2229] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:59.134846 12275 raft_consensus.cc:2258] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:59.134927 12276 raft_consensus.cc:2258] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:59.135185 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 11842
I20260923 22:17:59.135282 12275 ts_tablet_manager.cc:1905] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:59.135301 12276 ts_tablet_manager.cc:1905] T e1ecdb554e71426ca0351620252ee9d4 P 4c2688bd70e3411c85db0a588adde80a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:59.136780 12275 ts_tablet_manager.cc:1918] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:59.136826 12275 log.cc:1196] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/e1ecdb554e71426ca0351620252ee9d4
I20260923 22:17:59.137042 12275 ts_tablet_manager.cc:1939] T e1ecdb554e71426ca0351620252ee9d4 P 3c35e4ffb5d242dc91cd6ff0a22bfc55: Deleting consensus metadata
I20260923 22:17:59.137329 11765 catalog_manager.cc:4841] TS 3c35e4ffb5d242dc91cd6ff0a22bfc55 (127.1.129.66:34717): tablet e1ecdb554e71426ca0351620252ee9d4 (table test-table [id=00444ee820ec48039edf8f788ed9f332]) successfully deleted
I20260923 22:17:59.141439 12277 tablet_replica.cc:330] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: stopping tablet replica
I20260923 22:17:59.141567 12277 raft_consensus.cc:2229] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:59.141680 12277 raft_consensus.cc:2258] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 22:17:59.141757 11767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:38255 (error 108)
I20260923 22:17:59.141880 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 11976
I20260923 22:17:59.142741 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.142578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:38255 (local address 127.0.0.1:55728)
0923 22:17:59.142658 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:59.142702 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:38255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 22:17:59.142863 11767 catalog_manager.cc:4584] TS 4c2688bd70e3411c85db0a588adde80a (127.1.129.65:38255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1ecdb554e71426ca0351620252ee9d4: Network error: Client connection negotiation failed: client connection to 127.1.129.65:38255: connect: Connection refused (error 111)
I20260923 22:17:59.143908 12277 ts_tablet_manager.cc:1905] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:59.145397 12277 ts_tablet_manager.cc:1918] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:59.145581 12277 log.cc:1196] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/e1ecdb554e71426ca0351620252ee9d4
I20260923 22:17:59.145941 12277 ts_tablet_manager.cc:1939] T e1ecdb554e71426ca0351620252ee9d4 P 0174542109504565b898bcd6b89dfa71: Deleting consensus metadata
I20260923 22:17:59.146314 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 12110
I20260923 22:17:59.146400 11767 catalog_manager.cc:4841] TS 0174542109504565b898bcd6b89dfa71 (127.1.129.67:46193): tablet e1ecdb554e71426ca0351620252ee9d4 (table test-table [id=00444ee820ec48039edf8f788ed9f332]) successfully deleted
I20260923 22:17:59.151415 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 11748
2026-09-23T22:17:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2047 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:59 dist-test-slave-dmw7 krb5kdc[12281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:59 dist-test-slave-dmw7 krb5kdc[12281](info): set up 2 sockets
Sep 23 22:17:59 dist-test-slave-dmw7 krb5kdc[12281](info): commencing operation
krb5kdc: starting...
W20260923 22:18:00.187484 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:18:00 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:18:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.129.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.129.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/rangeradmin_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/rangeradmin_127.1.129.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.129.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.129.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/rangerlookup_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/rangerlookup_127.1.129.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.129.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.129.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/HTTP_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/HTTP_127.1.129.84@KRBTEST.COM.keytab.
I20260923 22:18:00.343014 1541 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDHvgV0/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/postgres -l logfile start
2026-09-23 22:18:02.755 UTC [12311] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 22:18:02.755 UTC [12311] LOG: listening on IPv4 address "127.1.129.84", port 46329
2026-09-23 22:18:02.759 UTC [12311] LOG: listening on Unix socket "/tmp/.s.PGSQL.46329"
2026-09-23 22:18:02.793 UTC [12314] LOG: database system was shut down at 2026-09-23 22:18:01 UTC
2026-09-23 22:18:02.798 UTC [12311] LOG: database system is ready to accept connections
I20260923 22:18:03.735466 1541 mini_postgres.cc:96] Postgres bound to 46329
2026-09-23 22:18:03.741 UTC [12322] FATAL: database "slave" does not exist
127.1.129.84:46329 - accepting connections
I20260923 22:18:03.741880 1541 mini_ranger.cc:161] Starting Ranger...
I20260923 22:18:03.756464 1541 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 22:18:04.256536 1541 mini_ranger.cc:87] Created ranger Postgres database
I20260923 22:18:04.256599 1541 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-admin
2026-09-23 22:18:04,536 [I] DB FLAVOR :POSTGRES
2026-09-23 22:18:04,537 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 22:18:04,537 [I] Checking connection..
2026-09-23 22:18:04,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 22:18:04,777 [I] Checking connection passed.
2026-09-23 22:18:04,777 [I] --------- Verifying version history table ---------
2026-09-23 22:18:04,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:05,038 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 22:18:05,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:05,273 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 22:18:05,273 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 22:18:05,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 22:18:05.510 UTC [12386] WARNING: there is no transaction in progress
2026-09-23 22:18:05,516 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 22:18:05,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:05,768 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 22:18:05,768 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 22:18:05,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 22:18:06,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 22:18:06,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dmw7','N') ;"
2026-09-23 22:18:06,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 22:18:06,737 [I] Table x_portal_user does not exist in database ranger
2026-09-23 22:18:06,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 22:18:06,989 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 22:18:06,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 22:18:07,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 22:18:07,488 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 22:18:07,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 22:18:08.446 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.464 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.481 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.497 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.513 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.532 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.592 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.600 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.615 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.626 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.635 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.650 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.660 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.666 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.673 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:08.680 UTC [12494] WARNING: there is no transaction in progress
2026-09-23 22:18:09,340 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 22:18:09,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 22:18:09,561 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 22:18:09,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 22:18:09,816 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 22:18:09,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 22:18:10,047 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 22:18:10,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 22:18:10,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 22:18:10,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dmw7';"
2026-09-23 22:18:10,730 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 22:18:10,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 22:18:10,989 [I] DB_PATCHES have already been applied
I20260923 22:18:10.995442 1541 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDHvgV0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 22:18:10.995517 1541 mini_ranger.cc:193] Using host: 127.1.129.84
I20260923 22:18:10.998618 1541 mini_ranger.cc:227] Ranger admin URL: http://127.1.129.84:35183
Sep 23, 2026 10:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 10:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 10:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 10:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 10:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 10:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 10:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 22:18:12.108934 1541 mini_ranger.cc:160] Time spent starting Ranger: real 8.367s user 0.003s sys 0.006s
Sep 23, 2026 10:18:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 10:18:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 10:18:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 10:18:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 10:18:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 22:18:32.959075 1541 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.129.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.129.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/rangerkms_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/rangerkms_127.1.129.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.129.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.129.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.129.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/HTTP_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.129.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/HTTP_127.1.129.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 22:18:33 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201913, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 22:18:33.042363 1541 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 22:18:33.051707 1541 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 22:18:33.958155 1541 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 22:18:33.958240 1541 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms
I20260923 22:18:33.958263 1541 mini_ranger_kms.cc:227] Using postgres at 127.1.129.84:46329
2026-09-23 22:18:34,136 [I] DB FLAVOR :POSTGRES
2026-09-23 22:18:34,137 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 22:18:34,137 [I] Checking connection
2026-09-23 22:18:34,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 22:18:34,384 [I] connection success
2026-09-23 22:18:34,384 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 22:18:34,385 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 22:18:34,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 22:18:34,624 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 22:18:34,624 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 22:18:34,624 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 22:18:34,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:46329/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 22:18:34,885 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 22:18:35.322743 1541 mini_ranger_kms.cc:310] Created kms service
I20260923 22:18:35.512989 1541 mini_ranger_kms.cc:326] Created kudu user
I20260923 22:18:35.558226 1541 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 22:18:35 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201913, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:18:35.949764 1541 mini_ranger_kms.cc:384] Added ranger policy
I20260923 22:18:35.949879 1541 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 22:18:35.949903 1541 mini_ranger_kms.cc:240] Using host: 127.1.129.84
I20260923 22:18:35.953788 1541 mini_ranger_kms.cc:276] Ranger KMS PID: 12674
I20260923 22:18:35.953910 1541 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.129.84:35233
Sep 23, 2026 10:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 10:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 10:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 10:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 10:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 10:18:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 22:18:38.283731 1541 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.241s user 0.000s sys 0.162s
I20260923 22:18:38.283838 1541 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 22:18:38.283915 1541 mini_ranger_kms.cc:401] 127.1.129.84:35233/kms/v1/keys
Sep 23 22:18:38 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201918, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.129.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 22:18:38 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201918, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.129.84@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
Sep 23 22:18:39 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201918, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.129.84@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
Sep 23 22:18:41 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:18:41.429399 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44779
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:44779
--ranger_config_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:35233/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:41.510329 12725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:41.510541 12725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:41.510572 12725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:41.511767 12725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:18:41.511812 12725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:41.511826 12725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:41.511839 12725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:18:41.511886 12725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:18:41.513408 12725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.129.84:35233/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:44779
--ranger_config_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:18:41.513648 12725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:41.513831 12725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:41.518568 12730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:41.518661 12731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:41.518581 12733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:41.518617 12725 server_base.cc:1034] running on GCE node
I20260923 22:18:41.519019 12725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:41.519325 12725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:41.520502 12725 hybrid_clock.cc:648] HybridClock initialized: now 1790201921520479 us; error 43 us; skew 500 ppm
Sep 23 22:18:41 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:41.527043 12725 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:18:41.527540 12725 webserver.cc:460] Webserver started at http://127.1.129.126:43321/ using document root <none> and password file <none>
I20260923 22:18:41.527762 12725 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:41.527823 12725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:41.527982 12725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:18:41 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:18:41.580629 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "c424495c5a3148a5b40e2384eab901b0"
format_stamp: "Formatted at 2026-09-23 22:18:41 on dist-test-slave-dmw7"
server_key: "8a13a14d8241fa2f967c75c3bdec9b91"
server_key_iv: "95a15cf90e481e0c5b6d146b244ec5f0"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:41.581058 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "c424495c5a3148a5b40e2384eab901b0"
format_stamp: "Formatted at 2026-09-23 22:18:41 on dist-test-slave-dmw7"
server_key: "8a13a14d8241fa2f967c75c3bdec9b91"
server_key_iv: "95a15cf90e481e0c5b6d146b244ec5f0"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:41.582571 12725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:18:41.594444 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:41.594631 12725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:18:41.594763 12725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "c424495c5a3148a5b40e2384eab901b0"
format_stamp: "Formatted at 2026-09-23 22:18:41 on dist-test-slave-dmw7"
server_key: "8a13a14d8241fa2f967c75c3bdec9b91"
server_key_iv: "95a15cf90e481e0c5b6d146b244ec5f0"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:41.594884 12725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:41.629132 12725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:41.631356 12725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:41.631562 12725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:41.636734 12725 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:44779
I20260923 22:18:41.636754 12792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:44779 every 8 connection(s)
I20260923 22:18:41.637254 12725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:18:41.638067 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:41.641052 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0: Bootstrap starting.
I20260923 22:18:41.641717 12793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:41.642027 12793 log.cc:824] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:41.642787 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0: No bootstrap required, opened a new log
I20260923 22:18:41.645601 12793 raft_consensus.cc:348] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c424495c5a3148a5b40e2384eab901b0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44779 } }
I20260923 22:18:41.645797 12793 raft_consensus.cc:374] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:41.645834 12793 raft_consensus.cc:729] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c424495c5a3148a5b40e2384eab901b0, State: Initialized, Role: FOLLOWER
I20260923 22:18:41.645980 12793 consensus_queue.cc:260] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c424495c5a3148a5b40e2384eab901b0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44779 } }
I20260923 22:18:41.646054 12793 raft_consensus.cc:388] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:18:41.646082 12793 raft_consensus.cc:482] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:18:41.646127 12793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:41.646836 12793 raft_consensus.cc:504] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c424495c5a3148a5b40e2384eab901b0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44779 } }
I20260923 22:18:41.646941 12793 leader_election.cc:302] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c424495c5a3148a5b40e2384eab901b0; no voters:
I20260923 22:18:41.647107 12793 leader_election.cc:288] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:18:41.647300 12797 sys_catalog.cc:455] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c424495c5a3148a5b40e2384eab901b0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44779 } } }
I20260923 22:18:41.647359 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 12725
I20260923 22:18:41.647445 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:18:41.647493 12797 sys_catalog.cc:458] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:18:41.647351 12793 sys_catalog.cc:564] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:18:41.647643 1541 external_mini_cluster.cc:1428] Setting key a0398b67a86bd005bc565fe997c6b1bb
I20260923 22:18:41.647854 12796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:18:41.647974 12796 raft_consensus.cc:686] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [term 1 LEADER]: Becoming Leader. State: Replica: c424495c5a3148a5b40e2384eab901b0, State: Running, Role: LEADER
I20260923 22:18:41.648119 12796 consensus_queue.cc:237] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c424495c5a3148a5b40e2384eab901b0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44779 } }
I20260923 22:18:41.648465 12797 sys_catalog.cc:455] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c424495c5a3148a5b40e2384eab901b0. Latest consensus state: current_term: 1 leader_uuid: "c424495c5a3148a5b40e2384eab901b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c424495c5a3148a5b40e2384eab901b0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44779 } } }
I20260923 22:18:41.648581 12797 sys_catalog.cc:458] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0 [sys.catalog]: This master's current role is: LEADER
I20260923 22:18:41.648988 12793 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 22:18:41.649380 12810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:18:41.652561 12810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:18:41.655413 12810 catalog_manager.cc:1347] Generated new cluster ID: e4b7d1031f874ad686e20d27223aef62
I20260923 22:18:41.655458 12810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:18:41 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:18:41.663158 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:41.648268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.1.129.126:44779)
0923 22:18:41.649122 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:41.649132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:41.649165 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:18:41.649415 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:41.649422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:41.649483 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:41.649626 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:41.651237 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:41.653931 (+ 2694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:41.655177 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:41.655872 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:41.655982 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:41.659028 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:41.659032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:41.659032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:41.659044 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:41.661211 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:41.661636 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:41.661639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:41.661639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:41.661746 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:41.662003 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:41.662004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:41.662005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:41.662341 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:41.662474 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:41.662778 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 22:18:41.662933 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":93,"threads_started":1}
I20260923 22:18:41.700210 12810 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:18:41.700965 12810 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:18:41.708943 12810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0: Generated new TSK 0
I20260923 22:18:41.709146 12810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:18:42.425624 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:35233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44779
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:42.533535 12837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:42.533771 12837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:42.533811 12837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:42.535019 12837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:42.535079 12837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:42.535121 12837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:18:42.536564 12837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.129.84:35233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:18:42.536816 12837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:42.537037 12837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:42.542912 12843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:42.542943 12845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:42.543162 12837 server_base.cc:1034] running on GCE node
W20260923 22:18:42.542943 12842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:42.543555 12837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:42.543833 12837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:42.544978 12837 hybrid_clock.cc:648] HybridClock initialized: now 1790201922544977 us; error 35 us; skew 500 ppm
Sep 23 22:18:42 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:42.547610 12837 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:18:42.548188 12837 webserver.cc:460] Webserver started at http://127.1.129.65:44255/ using document root <none> and password file <none>
I20260923 22:18:42.548485 12837 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:42.548556 12837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:42.548684 12837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:18:42 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:18:42.562986 12837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "deb05f7291a6492ca6183b542d0fd48c"
format_stamp: "Formatted at 2026-09-23 22:18:42 on dist-test-slave-dmw7"
server_key: "9e119ddb78d7b11d0ea4b09e90534ce0"
server_key_iv: "09a19bf1b7e138ef00accab93be808e8"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:42.563596 12837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "deb05f7291a6492ca6183b542d0fd48c"
format_stamp: "Formatted at 2026-09-23 22:18:42 on dist-test-slave-dmw7"
server_key: "9e119ddb78d7b11d0ea4b09e90534ce0"
server_key_iv: "09a19bf1b7e138ef00accab93be808e8"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:42.566962 12837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260923 22:18:42.579985 12854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:42.580219 12837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:18:42.580340 12837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "deb05f7291a6492ca6183b542d0fd48c"
format_stamp: "Formatted at 2026-09-23 22:18:42 on dist-test-slave-dmw7"
server_key: "9e119ddb78d7b11d0ea4b09e90534ce0"
server_key_iv: "09a19bf1b7e138ef00accab93be808e8"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:42.580439 12837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:42.607656 12837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:42.610293 12837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:42.610472 12837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:42.610765 12837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:42.611138 12837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:42.611189 12837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:42.611222 12837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:42.611281 12837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:42.619401 12837 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39121
I20260923 22:18:42.619494 12967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39121 every 8 connection(s)
I20260923 22:18:42.619827 12837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:18:42.621977 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 12837
I20260923 22:18:42.622092 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:18:42.622283 1541 external_mini_cluster.cc:1428] Setting key b43bb7f152fd9b37248e9ab4ba7966ca
Sep 23 22:18:42 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:18:42.632772 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:42.620737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:49807 (local address 127.1.129.126:44779)
0923 22:18:42.620885 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:42.620887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:42.620987 (+ 100us) server_negotiation.cc:407] Connection header received
0923 22:18:42.621647 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:42.621648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:42.621659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:42.621730 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:42.622416 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:42.623240 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:42.624716 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:42.625034 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:42.625109 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:42.627793 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:42.627795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:42.627795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:42.627804 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:42.629602 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:42.630121 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:42.630123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:42.630123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:42.630189 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:42.630478 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:42.630479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:42.630479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:42.630613 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:42.630713 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:42.632571 (+ 1858us) server_negotiation.cc:299] Negotiation successful
0923 22:18:42.632658 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260923 22:18:42.633530 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:42.620647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44779 (local address 127.1.129.65:49807)
0923 22:18:42.620840 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:42.620890 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:42.621536 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:42.621802 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:42.621807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:42.622031 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:42.622294 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:42.622310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:42.623361 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:42.623362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:42.624631 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:42.624634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:42.625138 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:42.625139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:42.625284 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:42.625969 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:42.625978 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:42.627695 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:42.629710 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:42.629716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:42.629719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:42.630038 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:42.630273 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:42.630273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:42.630274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:42.630407 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:42.632215 (+ 1808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:42.632240 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:42.632400 (+ 160us) client_negotiation.cc:769] Sending connection context
0923 22:18:42.633133 (+ 733us) client_negotiation.cc:240] Negotiation successful
0923 22:18:42.633301 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260923 22:18:42.635637 12968 heartbeater.cc:347] Connected to a master server at 127.1.129.126:44779
I20260923 22:18:42.635730 12968 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:42.635929 12968 heartbeater.cc:511] Master 127.1.129.126:44779 requested a full tablet report, sending...
I20260923 22:18:42.636507 12757 ts_manager.cc:194] Registered new tserver with Master: deb05f7291a6492ca6183b542d0fd48c (127.1.129.65:39121)
I20260923 22:18:42.637185 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:49807
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:18:42.665668 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:35233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44779
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:18:42 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 22:18:42.762322 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:42.762553 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:42.762593 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:42.763959 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:42.764057 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:42.764115 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:18:42.765833 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.129.84:35233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:18:42.766224 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:42.766535 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:42.773686 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:42.773751 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:42.773713 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:42.774566 12976 server_base.cc:1034] running on GCE node
I20260923 22:18:42.774804 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:42.775085 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:42.776278 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790201922776242 us; error 40 us; skew 500 ppm
Sep 23 22:18:42 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:42.779446 12976 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:18:42.779989 12976 webserver.cc:460] Webserver started at http://127.1.129.66:33875/ using document root <none> and password file <none>
I20260923 22:18:42.780251 12976 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:42.780311 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:42.780442 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:18:42 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:18:42.798961 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "d3f38e7359a14e1e828f8d30901216b9"
format_stamp: "Formatted at 2026-09-23 22:18:42 on dist-test-slave-dmw7"
server_key: "7cb7b84e060e6fddf56fc284a0bc643c"
server_key_iv: "02b13fdc0adabcbabe4721f02a2b7f6b"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:42.799439 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "d3f38e7359a14e1e828f8d30901216b9"
format_stamp: "Formatted at 2026-09-23 22:18:42 on dist-test-slave-dmw7"
server_key: "7cb7b84e060e6fddf56fc284a0bc643c"
server_key_iv: "02b13fdc0adabcbabe4721f02a2b7f6b"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:42.801368 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260923 22:18:42.816882 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:42.817088 12976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 22:18:42.817180 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "d3f38e7359a14e1e828f8d30901216b9"
format_stamp: "Formatted at 2026-09-23 22:18:42 on dist-test-slave-dmw7"
server_key: "7cb7b84e060e6fddf56fc284a0bc643c"
server_key_iv: "02b13fdc0adabcbabe4721f02a2b7f6b"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:42.817294 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:42.832440 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:42.839237 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:42.839424 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:42.839697 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:42.840121 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:42.840210 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:42.840276 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:42.840324 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:42.848296 12976 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:33385
I20260923 22:18:42.848991 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:18:42.852861 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 12976
I20260923 22:18:42.853179 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:18:42.853549 1541 external_mini_cluster.cc:1428] Setting key 569d92642c2445f7df45e8ae8a964e16
I20260923 22:18:42.865760 13105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:33385 every 8 connection(s)
Sep 23 22:18:42 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:18:42.875850 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:42.851376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44779 (local address 127.1.129.66:58909)
0923 22:18:42.851852 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:42.851943 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:42.855213 (+ 3270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:42.857153 (+ 1940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:42.857159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:42.857411 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:42.857702 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:42.857709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:42.861018 (+ 3309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:42.861020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:42.862371 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:42.862375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:42.863872 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:42.863873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:42.864056 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:42.864707 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:42.864719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:42.869355 (+ 4636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:42.871486 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:42.871499 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:42.871503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:42.871876 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:42.872151 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:42.872152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:42.872152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:42.872289 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:42.875094 (+ 2805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:42.875102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:42.875285 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 22:18:42.875404 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 22:18:42.875581 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":119,"threads_started":1}
I20260923 22:18:42.876845 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:42.851642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:58909 (local address 127.1.129.126:44779)
0923 22:18:42.851703 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:42.851705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:42.852043 (+ 338us) server_negotiation.cc:407] Connection header received
0923 22:18:42.856933 (+ 4890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:42.856934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:42.856949 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:42.857024 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:42.857813 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:42.858775 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:42.862488 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:42.863704 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:42.863788 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:42.869471 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:42.869473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:42.869473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:42.869483 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:42.871397 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:42.872010 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:42.872011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:42.872012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:42.872076 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:42.874840 (+ 2764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:42.874841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:42.874842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:42.874997 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:42.875056 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:42.876639 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0923 22:18:42.876741 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:18:42.877301 13106 heartbeater.cc:347] Connected to a master server at 127.1.129.126:44779
I20260923 22:18:42.877400 13106 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:42.877600 13106 heartbeater.cc:511] Master 127.1.129.126:44779 requested a full tablet report, sending...
I20260923 22:18:42.883263 12757 ts_manager.cc:194] Registered new tserver with Master: d3f38e7359a14e1e828f8d30901216b9 (127.1.129.66:33385)
I20260923 22:18:42.883836 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:58909
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:18:42.925146 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:35233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44779
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:43.065356 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:43.065642 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:43.065680 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:43.067862 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:43.067930 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:43.067973 13111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:18:43.070302 13111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.129.84:35233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:18:43.070621 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:43.070925 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:43.083418 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:43.083429 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:43.083544 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:43.084650 13111 server_base.cc:1034] running on GCE node
I20260923 22:18:43.084887 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:43.085156 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:43.086329 13111 hybrid_clock.cc:648] HybridClock initialized: now 1790201923086303 us; error 39 us; skew 500 ppm
Sep 23 22:18:43 dist-test-slave-dmw7 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:43.090000 13111 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:18:43.090544 13111 webserver.cc:460] Webserver started at http://127.1.129.67:36503/ using document root <none> and password file <none>
I20260923 22:18:43.090870 13111 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:43.090937 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:43.091073 13111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:18:43 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:18:43.107722 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "9a3036a02d7d40e59f033e0a3e8e4525"
format_stamp: "Formatted at 2026-09-23 22:18:43 on dist-test-slave-dmw7"
server_key: "f8f2aafb558ddfb926a0736d88d1d75f"
server_key_iv: "b4840956cc6643740cd863a28a98f427"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:43.108166 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "9a3036a02d7d40e59f033e0a3e8e4525"
format_stamp: "Formatted at 2026-09-23 22:18:43 on dist-test-slave-dmw7"
server_key: "f8f2aafb558ddfb926a0736d88d1d75f"
server_key_iv: "b4840956cc6643740cd863a28a98f427"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:43.110448 13111 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260923 22:18:43.132450 13126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:43.132649 13111 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260923 22:18:43.132741 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "9a3036a02d7d40e59f033e0a3e8e4525"
format_stamp: "Formatted at 2026-09-23 22:18:43 on dist-test-slave-dmw7"
server_key: "f8f2aafb558ddfb926a0736d88d1d75f"
server_key_iv: "b4840956cc6643740cd863a28a98f427"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:43.132843 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:43.152978 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:43.155313 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:43.155462 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:43.155663 13111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:43.155987 13111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:43.156026 13111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:43.156062 13111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:43.156080 13111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:43.162523 13111 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:42613
I20260923 22:18:43.162961 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:18:43.170770 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 13111
I20260923 22:18:43.170908 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:18:43.171126 1541 external_mini_cluster.cc:1428] Setting key d2d880d17fa7f5930c8a5947a2fbfd75
I20260923 22:18:43.175244 13239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:42613 every 8 connection(s)
Sep 23 22:18:43 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:18:43.191246 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.176132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:36777 (local address 127.1.129.126:44779)
0923 22:18:43.176239 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:43.176242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:43.179611 (+ 3369us) server_negotiation.cc:407] Connection header received
0923 22:18:43.181557 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:43.181559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:43.181573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:43.181606 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:43.182486 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.183220 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.185143 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.185381 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.185427 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.188114 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:43.188115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:43.188116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:43.188122 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:43.189720 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:43.190199 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:43.190200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:43.190200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:43.190252 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:43.190516 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:43.190517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:43.190517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:43.190673 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:43.190714 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:43.191064 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 22:18:43.191149 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 22:18:43.191774 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.175958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44779 (local address 127.1.129.67:36777)
0923 22:18:43.179433 (+ 3475us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:43.179504 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:43.180784 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:43.181811 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:43.181816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:43.182071 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:43.182357 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.182364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.183872 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.183873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.185050 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.185053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.185528 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.185529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.185663 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.186333 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:43.186343 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:43.187994 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:43.189807 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:43.189813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:43.189815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:43.190113 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:43.190318 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:43.190318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:43.190319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:43.190446 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:43.190757 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:43.190763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:43.190921 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 22:18:43.191382 (+ 461us) client_negotiation.cc:240] Negotiation successful
0923 22:18:43.191549 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3410,"thread_start_us":115,"threads_started":1}
I20260923 22:18:43.192323 13240 heartbeater.cc:347] Connected to a master server at 127.1.129.126:44779
I20260923 22:18:43.192422 13240 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:43.192615 13240 heartbeater.cc:511] Master 127.1.129.126:44779 requested a full tablet report, sending...
I20260923 22:18:43.192940 12757 ts_manager.cc:194] Registered new tserver with Master: 9a3036a02d7d40e59f033e0a3e8e4525 (127.1.129.67:42613)
I20260923 22:18:43.193405 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:36777
I20260923 22:18:43.194449 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:18:43.203884 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.196346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.1.129.126:44779)
0923 22:18:43.196391 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:43.196392 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:43.196715 (+ 323us) server_negotiation.cc:407] Connection header received
0923 22:18:43.196801 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:43.196801 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:43.196810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:43.196837 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:43.197574 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.198229 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.199322 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.199618 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.199658 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.201119 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:43.201120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:43.201121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:43.201127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:43.202769 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:43.203126 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:43.203127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:43.203128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:43.203176 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:43.203357 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:43.203357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:43.203358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:43.203474 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:43.203540 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:43.203730 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 22:18:43.203800 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:18:43.205699 12757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:18:43.207849 12757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:18:43.225049 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.210951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:42613 (local address 127.0.0.1:34310)
0923 22:18:43.214719 (+ 3768us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:43.214731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:43.214793 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:43.220760 (+ 5967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:43.220761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:43.220764 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:43.220807 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.220808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.222408 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.222409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.223604 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.223607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.224719 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.224720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.224821 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.224837 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 22:18:43.224904 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 22:18:43.224957 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3748,"mutex_wait_us":30,"thread_start_us":59,"threads_started":1}
I20260923 22:18:43.226397 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.211263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.1.129.67:42613)
0923 22:18:43.211925 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:43.211929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:43.214878 (+ 2949us) server_negotiation.cc:407] Connection header received
0923 22:18:43.214923 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:43.214926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:43.214942 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:43.215011 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:43.220902 (+ 5891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.222315 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.223753 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.224516 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.224598 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.224681 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:43.226261 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0923 22:18:43.226309 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":79,"threads_started":1}
I20260923 22:18:43.227201 13174 tablet_service.cc:1467] Processing CreateTablet for tablet daea74c5f4404775ae3451f52fe9a0ca (DEFAULT_TABLE table=test-table [id=79c5a2dd940343de87b61347b1bea3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:43.227589 13174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daea74c5f4404775ae3451f52fe9a0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:43.234436 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.210656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:33385 (local address 127.0.0.1:49482)
0923 22:18:43.211703 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:43.211743 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:43.211797 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:43.215314 (+ 3517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:43.215317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:43.215319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:43.215385 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.215387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.218577 (+ 3190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.218578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.219918 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.219921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.234110 (+ 14189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.234112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.234186 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.234193 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 22:18:43.234225 (+ 32us) client_negotiation.cc:240] Negotiation successful
0923 22:18:43.234351 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":62,"threads_started":1}
I20260923 22:18:43.234964 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.210880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.1.129.66:33385)
0923 22:18:43.215152 (+ 4272us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:43.215157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:43.215167 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:18:43.215201 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:43.215203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:43.215218 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:43.215464 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:43.215536 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.217032 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.220072 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.234006 (+ 13934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.234687 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.234806 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:43.234838 (+ 32us) server_negotiation.cc:299] Negotiation successful
0923 22:18:43.234888 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4234,"thread_start_us":90,"threads_started":1}
I20260923 22:18:43.235659 13259 tablet_bootstrap.cc:492] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: Bootstrap starting.
I20260923 22:18:43.235597 13040 tablet_service.cc:1467] Processing CreateTablet for tablet daea74c5f4404775ae3451f52fe9a0ca (DEFAULT_TABLE table=test-table [id=79c5a2dd940343de87b61347b1bea3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:43.236008 13040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daea74c5f4404775ae3451f52fe9a0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:43.236665 13259 tablet_bootstrap.cc:654] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:43.236960 13259 log.cc:824] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:43.240204 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.211094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39121 (local address 127.0.0.1:47928)
0923 22:18:43.214740 (+ 3646us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:43.214751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:43.214793 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:43.217140 (+ 2347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:43.217141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:43.217144 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:43.217184 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.217185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.222670 (+ 5485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.222671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.232799 (+ 10128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.232803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.239955 (+ 7152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.239956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.240039 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.240044 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:18:43.240075 (+ 31us) client_negotiation.cc:240] Negotiation successful
0923 22:18:43.240123 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3632,"thread_start_us":48,"threads_started":1}
I20260923 22:18:43.240635 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.211198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.1.129.65:39121)
0923 22:18:43.215849 (+ 4651us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:43.215854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:43.215863 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:18:43.215895 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:43.215897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:43.215913 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:43.217253 (+ 1340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:43.217317 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.222534 (+ 5217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.232979 (+ 10445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.233824 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.240386 (+ 6562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.240481 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:43.240510 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 22:18:43.240565 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4616,"thread_start_us":55,"threads_started":1}
I20260923 22:18:43.241044 13261 tablet_bootstrap.cc:492] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Bootstrap starting.
I20260923 22:18:43.241305 12902 tablet_service.cc:1467] Processing CreateTablet for tablet daea74c5f4404775ae3451f52fe9a0ca (DEFAULT_TABLE table=test-table [id=79c5a2dd940343de87b61347b1bea3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:43.241653 12902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daea74c5f4404775ae3451f52fe9a0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:43.241997 13261 tablet_bootstrap.cc:654] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:43.242293 13261 log.cc:824] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:43.243019 13261 tablet_bootstrap.cc:492] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: No bootstrap required, opened a new log
I20260923 22:18:43.243086 13261 ts_tablet_manager.cc:1397] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:18:43.244971 13261 raft_consensus.cc:348] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.245139 13261 raft_consensus.cc:374] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:43.245175 13261 raft_consensus.cc:729] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f38e7359a14e1e828f8d30901216b9, State: Initialized, Role: FOLLOWER
I20260923 22:18:43.245280 13261 consensus_queue.cc:260] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.245514 13261 ts_tablet_manager.cc:1428] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:18:43.245651 13106 heartbeater.cc:503] Master 127.1.129.126:44779 was elected leader, sending a full tablet report...
I20260923 22:18:43.247211 13259 tablet_bootstrap.cc:492] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: No bootstrap required, opened a new log
I20260923 22:18:43.247320 13259 ts_tablet_manager.cc:1397] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260923 22:18:43.249975 13259 raft_consensus.cc:348] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.250173 13259 raft_consensus.cc:374] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:43.250211 13259 raft_consensus.cc:729] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3036a02d7d40e59f033e0a3e8e4525, State: Initialized, Role: FOLLOWER
I20260923 22:18:43.250360 13259 consensus_queue.cc:260] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.250651 13259 ts_tablet_manager.cc:1428] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 22:18:43.250766 13240 heartbeater.cc:503] Master 127.1.129.126:44779 was elected leader, sending a full tablet report...
I20260923 22:18:43.254957 13263 tablet_bootstrap.cc:492] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c: Bootstrap starting.
I20260923 22:18:43.255779 13263 tablet_bootstrap.cc:654] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:43.256145 13263 log.cc:824] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:43.256929 13263 tablet_bootstrap.cc:492] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c: No bootstrap required, opened a new log
I20260923 22:18:43.256991 13263 ts_tablet_manager.cc:1397] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:18:43.259521 13263 raft_consensus.cc:348] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.259677 13263 raft_consensus.cc:374] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:43.259706 13263 raft_consensus.cc:729] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb05f7291a6492ca6183b542d0fd48c, State: Initialized, Role: FOLLOWER
I20260923 22:18:43.260361 13263 consensus_queue.cc:260] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.260707 13263 ts_tablet_manager.cc:1428] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 22:18:43.260900 12968 heartbeater.cc:503] Master 127.1.129.126:44779 was elected leader, sending a full tablet report...
Sep 23 22:18:43 dist-test-slave-dmw7 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
W20260923 22:18:43.350843 13107 tablet.cc:2369] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:18:43.363521 13265 raft_consensus.cc:482] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:18:43.363653 13265 raft_consensus.cc:504] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.364037 13265 leader_election.cc:288] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3f38e7359a14e1e828f8d30901216b9 (127.1.129.66:33385), deb05f7291a6492ca6183b542d0fd48c (127.1.129.65:39121)
I20260923 22:18:43.367758 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.364452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:33217 (local address 127.1.129.66:33385)
0923 22:18:43.364493 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:43.364495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:43.364502 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:18:43.364529 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:43.364530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:43.364541 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:43.364586 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:43.364792 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.365543 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.366931 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.367372 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.367473 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.367483 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:43.367651 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 22:18:43.367692 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:18:43.367758 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.364203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:33385 (local address 127.1.129.67:33217)
0923 22:18:43.364323 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:43.364335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:43.364375 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:43.364634 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:43.364635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:43.364637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:43.364707 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.364708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.365712 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.365714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.366830 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.366833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.367469 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.367470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.367551 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.367555 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:18:43.367644 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 22:18:43.367682 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260923 22:18:43.368124 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daea74c5f4404775ae3451f52fe9a0ca" candidate_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f38e7359a14e1e828f8d30901216b9" is_pre_election: true
I20260923 22:18:43.368317 13060 raft_consensus.cc:2454] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3036a02d7d40e59f033e0a3e8e4525 in term 0.
I20260923 22:18:43.368547 13127 leader_election.cc:302] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3036a02d7d40e59f033e0a3e8e4525, d3f38e7359a14e1e828f8d30901216b9; no voters:
I20260923 22:18:43.368696 13265 raft_consensus.cc:2781] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:18:43.368741 13265 raft_consensus.cc:482] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:18:43.368763 13265 raft_consensus.cc:3037] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:43.369647 13265 raft_consensus.cc:504] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.369782 13265 leader_election.cc:288] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [CANDIDATE]: Term 1 election: Requested vote from peers d3f38e7359a14e1e828f8d30901216b9 (127.1.129.66:33385), deb05f7291a6492ca6183b542d0fd48c (127.1.129.65:39121)
I20260923 22:18:43.369961 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daea74c5f4404775ae3451f52fe9a0ca" candidate_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f38e7359a14e1e828f8d30901216b9"
I20260923 22:18:43.370061 13060 raft_consensus.cc:3037] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 0 FOLLOWER]: Advancing to term 1
W20260923 22:18:43.370790 12969 tablet.cc:2369] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:18:43.370976 13060 raft_consensus.cc:2454] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3036a02d7d40e59f033e0a3e8e4525 in term 1.
I20260923 22:18:43.371141 13127 leader_election.cc:302] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3036a02d7d40e59f033e0a3e8e4525, d3f38e7359a14e1e828f8d30901216b9; no voters:
I20260923 22:18:43.371225 13265 raft_consensus.cc:2781] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:18:43.371378 13265 raft_consensus.cc:686] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3036a02d7d40e59f033e0a3e8e4525, State: Running, Role: LEADER
I20260923 22:18:43.371513 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.364650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:50565 (local address 127.1.129.65:39121)
0923 22:18:43.364692 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:43.364693 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:43.367963 (+ 3270us) server_negotiation.cc:407] Connection header received
0923 22:18:43.367991 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:43.367992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:43.368003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:43.368047 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:43.368184 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.368840 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.370014 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.370553 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.371137 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.371147 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:43.371410 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 22:18:43.371454 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:18:43.371484 13265 consensus_queue.cc:237] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } }
I20260923 22:18:43.372344 12757 catalog_manager.cc:5495] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3036a02d7d40e59f033e0a3e8e4525 (127.1.129.67). New cstate: current_term: 1 leader_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 } health_report { overall_health: UNKNOWN } } }
I20260923 22:18:43.372733 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.364475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39121 (local address 127.1.129.67:50565)
0923 22:18:43.367844 (+ 3369us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:43.367856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:43.367904 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:43.368088 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:43.368089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:43.368090 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:43.368118 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.368120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.368931 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.368932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.369907 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:43.369909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.371177 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.371178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:43.371239 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.371243 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:18:43.371296 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:18:43.371335 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3357,"thread_start_us":78,"threads_started":1}
I20260923 22:18:43.373055 12922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daea74c5f4404775ae3451f52fe9a0ca" candidate_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb05f7291a6492ca6183b542d0fd48c" is_pre_election: true
I20260923 22:18:43.373198 12922 raft_consensus.cc:2454] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3036a02d7d40e59f033e0a3e8e4525 in term 0.
I20260923 22:18:43.373333 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daea74c5f4404775ae3451f52fe9a0ca" candidate_uuid: "9a3036a02d7d40e59f033e0a3e8e4525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb05f7291a6492ca6183b542d0fd48c"
I20260923 22:18:43.373401 12921 raft_consensus.cc:3037] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:43.374038 12921 raft_consensus.cc:2454] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3036a02d7d40e59f033e0a3e8e4525 in term 1.
I20260923 22:18:43.402194 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.399150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.1.129.67:42613)
0923 22:18:43.399217 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:43.399219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:43.399228 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:18:43.399269 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:43.399270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:43.399299 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:43.399379 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:18:43.399525 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.400225 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.401258 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:43.401568 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:43.401634 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:43.401827 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:18:43.401956 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:18:43.402031 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:43.402082 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 22:18:43.402102 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:18:43.404595 12921 raft_consensus.cc:1261] T daea74c5f4404775ae3451f52fe9a0ca P deb05f7291a6492ca6183b542d0fd48c [term 1 FOLLOWER]: Refusing update from remote peer 9a3036a02d7d40e59f033e0a3e8e4525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:18:43.404659 13060 raft_consensus.cc:1261] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 1 FOLLOWER]: Refusing update from remote peer 9a3036a02d7d40e59f033e0a3e8e4525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:18:43.404964 13269 consensus_queue.cc:1035] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb05f7291a6492ca6183b542d0fd48c" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:18:43.405103 13265 consensus_queue.cc:1035] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f38e7359a14e1e828f8d30901216b9" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 33385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:18:43.408257 13273 mvcc.cc:205] Tried to move back new op lower bound from 7332667078262226944 to 7332667078130290688. Current Snapshot: MvccSnapshot[applied={T|T < 7332667078262226944}]
I20260923 22:18:43.408571 13272 mvcc.cc:205] Tried to move back new op lower bound from 7332667078262226944 to 7332667078130290688. Current Snapshot: MvccSnapshot[applied={T|T < 7332667078262226944}]
I20260923 22:18:43.413079 12757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:18:43.413188 12757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:18:43.413864 12757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c424495c5a3148a5b40e2384eab901b0: Sending DeleteTablet for 3 replicas of tablet daea74c5f4404775ae3451f52fe9a0ca
I20260923 22:18:43.414232 13040 tablet_service.cc:1514] Processing DeleteTablet for tablet daea74c5f4404775ae3451f52fe9a0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:43 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:49482
I20260923 22:18:43.414227 12902 tablet_service.cc:1514] Processing DeleteTablet for tablet daea74c5f4404775ae3451f52fe9a0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:43 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:47928
I20260923 22:18:43.414496 13174 tablet_service.cc:1514] Processing DeleteTablet for tablet daea74c5f4404775ae3451f52fe9a0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:43 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:34310
I20260923 22:18:43.414712 13284 tablet_replica.cc:330] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: stopping tablet replica
I20260923 22:18:43.414846 13284 raft_consensus.cc:2229] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:18:43.415012 13284 raft_consensus.cc:2258] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:18:43.415063 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 12837
I20260923 22:18:43.415483 13284 ts_tablet_manager.cc:1905] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:18:43.417591 13284 ts_tablet_manager.cc:1918] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:18:43.417716 13284 log.cc:1196] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/daea74c5f4404775ae3451f52fe9a0ca
I20260923 22:18:43.418056 13284 ts_tablet_manager.cc:1939] T daea74c5f4404775ae3451f52fe9a0ca P 9a3036a02d7d40e59f033e0a3e8e4525: Deleting consensus metadata
I20260923 22:18:43.418299 13283 tablet_replica.cc:330] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: stopping tablet replica
I20260923 22:18:43.418484 13283 raft_consensus.cc:2229] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:18:43.418551 12744 catalog_manager.cc:4841] TS 9a3036a02d7d40e59f033e0a3e8e4525 (127.1.129.67:42613): tablet daea74c5f4404775ae3451f52fe9a0ca (table test-table [id=79c5a2dd940343de87b61347b1bea3fa]) successfully deleted
I20260923 22:18:43.418671 13283 raft_consensus.cc:2258] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:18:43.419143 13283 ts_tablet_manager.cc:1905] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:18:43.421177 13283 ts_tablet_manager.cc:1918] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:18:43.421326 13283 log.cc:1196] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/daea74c5f4404775ae3451f52fe9a0ca
I20260923 22:18:43.421646 13283 ts_tablet_manager.cc:1939] T daea74c5f4404775ae3451f52fe9a0ca P d3f38e7359a14e1e828f8d30901216b9: Deleting consensus metadata
I20260923 22:18:43.422062 12741 catalog_manager.cc:4841] TS d3f38e7359a14e1e828f8d30901216b9 (127.1.129.66:33385): tablet daea74c5f4404775ae3451f52fe9a0ca (table test-table [id=79c5a2dd940343de87b61347b1bea3fa]) successfully deleted
W20260923 22:18:43.422607 12744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:39121 (error 108)
I20260923 22:18:43.422884 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 12976
I20260923 22:18:43.428285 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:43.428150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39121 (local address 127.0.0.1:47944)
0923 22:18:43.428207 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:43.428244 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 22:18:43.428460 12744 catalog_manager.cc:4584] TS deb05f7291a6492ca6183b542d0fd48c (127.1.129.65:39121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet daea74c5f4404775ae3451f52fe9a0ca: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39121: connect: Connection refused (error 111)
I20260923 22:18:43.428642 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 13111
I20260923 22:18:43.434857 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 12725
I20260923 22:18:43.439360 1541 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 22:18:43.784164 1541 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 22:18:43.784273 1541 mini_ranger.cc:66] Stopping Ranger...
I20260923 22:18:44.129133 1541 mini_ranger.cc:68] Stopped Ranger
2026-09-23 22:18:44.129 UTC [12311] LOG: received smart shutdown request
2026-09-23 22:18:44.134 UTC [12311] LOG: background worker "logical replication launcher" (PID 12320) exited with exit code 1
2026-09-23 22:18:44.135 UTC [12315] LOG: shutting down
2026-09-23 22:18:44.313 UTC [12311] LOG: database system is shut down
2026-09-23T22:18:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45271 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:18:44 dist-test-slave-dmw7 krb5kdc[13294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:18:44 dist-test-slave-dmw7 krb5kdc[13294](info): set up 2 sockets
Sep 23 22:18:44 dist-test-slave-dmw7 krb5kdc[13294](info): commencing operation
krb5kdc: starting...
W20260923 22:18:45.462818 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:18:45 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:18:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.129.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.129.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/rangeradmin_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/rangeradmin_127.1.129.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.129.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.129.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/rangerlookup_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/rangerlookup_127.1.129.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.129.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.129.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/HTTP_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/HTTP_127.1.129.84@KRBTEST.COM.keytab.
I20260923 22:18:45.629352 1541 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDHvgV0/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/postgres -l logfile start
2026-09-23 22:18:48.151 UTC [13324] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 22:18:48.151 UTC [13324] LOG: listening on IPv4 address "127.1.129.84", port 50809
2026-09-23 22:18:48.154 UTC [13324] LOG: listening on Unix socket "/tmp/.s.PGSQL.50809"
2026-09-23 22:18:48.189 UTC [13327] LOG: database system was shut down at 2026-09-23 22:18:46 UTC
2026-09-23 22:18:48.193 UTC [13324] LOG: database system is ready to accept connections
I20260923 22:18:49.134472 1541 mini_postgres.cc:96] Postgres bound to 50809
2026-09-23 22:18:49.140 UTC [13335] FATAL: database "slave" does not exist
127.1.129.84:50809 - accepting connections
I20260923 22:18:49.140748 1541 mini_ranger.cc:161] Starting Ranger...
I20260923 22:18:49.154759 1541 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 22:18:49.625759 1541 mini_ranger.cc:87] Created ranger Postgres database
I20260923 22:18:49.625856 1541 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-admin
2026-09-23 22:18:49,919 [I] DB FLAVOR :POSTGRES
2026-09-23 22:18:49,920 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 22:18:49,920 [I] Checking connection..
2026-09-23 22:18:49,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 22:18:50,161 [I] Checking connection passed.
2026-09-23 22:18:50,161 [I] --------- Verifying version history table ---------
2026-09-23 22:18:50,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:50,387 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 22:18:50,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:50,624 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 22:18:50,625 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 22:18:50,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 22:18:50.861 UTC [13399] WARNING: there is no transaction in progress
2026-09-23 22:18:50,868 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 22:18:50,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:51,097 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 22:18:51,097 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 22:18:51,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 22:18:51,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 22:18:51,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dmw7','N') ;"
2026-09-23 22:18:51,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 22:18:52,085 [I] Table x_portal_user does not exist in database ranger
2026-09-23 22:18:52,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 22:18:52,320 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 22:18:52,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 22:18:52,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 22:18:52,815 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 22:18:52,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 22:18:53.770 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.786 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.801 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.817 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.832 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.850 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.916 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.922 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.937 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.947 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.958 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.973 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.983 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.990 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:53.997 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:54.004 UTC [13507] WARNING: there is no transaction in progress
2026-09-23 22:18:54,682 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 22:18:54,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 22:18:54,933 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 22:18:54,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 22:18:55,189 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 22:18:55,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 22:18:55,428 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 22:18:55,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 22:18:55,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 22:18:55,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dmw7';"
2026-09-23 22:18:56,150 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 22:18:56,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 22:18:56,383 [I] DB_PATCHES have already been applied
I20260923 22:18:56.388827 1541 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDHvgV0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 22:18:56.388900 1541 mini_ranger.cc:193] Using host: 127.1.129.84
I20260923 22:18:56.391988 1541 mini_ranger.cc:227] Ranger admin URL: http://127.1.129.84:54545
Sep 23, 2026 10:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 10:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 10:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 10:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 10:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 10:18:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 10:18:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 22:18:57.512125 1541 mini_ranger.cc:160] Time spent starting Ranger: real 8.371s user 0.000s sys 0.009s
Sep 23, 2026 10:19:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 10:19:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 10:19:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 10:19:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 10:19:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 22:19:19.434360 1541 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.129.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.129.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/rangerkms_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.129.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/rangerkms_127.1.129.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.129.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.129.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.129.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/HTTP_127.1.129.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.129.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/HTTP_127.1.129.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 22:19:19 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201959, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 22:19:19.528607 1541 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 22:19:19.538125 1541 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 22:19:20.427392 1541 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 22:19:20.427488 1541 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms
I20260923 22:19:20.427507 1541 mini_ranger_kms.cc:227] Using postgres at 127.1.129.84:50809
2026-09-23 22:19:20,590 [I] DB FLAVOR :POSTGRES
2026-09-23 22:19:20,590 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 22:19:20,590 [I] Checking connection
2026-09-23 22:19:20,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 22:19:20,851 [I] connection success
2026-09-23 22:19:20,851 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 22:19:20,851 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 22:19:20,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 22:19:21,093 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 22:19:21,093 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 22:19:21,094 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 22:19:21,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.129.84:50809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 22:19:21,343 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 22:19:21.803644 1541 mini_ranger_kms.cc:310] Created kms service
I20260923 22:19:22.019698 1541 mini_ranger_kms.cc:326] Created kudu user
I20260923 22:19:22.066444 1541 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 22:19:22 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201959, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:22.473124 1541 mini_ranger_kms.cc:384] Added ranger policy
I20260923 22:19:22.473316 1541 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDHvgV0/build/release/bin/postgresql.jar:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDHvgV0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 22:19:22.473335 1541 mini_ranger_kms.cc:240] Using host: 127.1.129.84
I20260923 22:19:22.476806 1541 mini_ranger_kms.cc:276] Ranger KMS PID: 13687
I20260923 22:19:22.476884 1541 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.129.84:58735
Sep 23, 2026 10:19:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 10:19:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 10:19:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 10:19:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 10:19:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 10:19:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 22:19:24.750476 1541 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.222s user 0.000s sys 0.145s
I20260923 22:19:24.750579 1541 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 22:19:24.750624 1541 mini_ranger_kms.cc:401] 127.1.129.84:58735/kms/v1/keys
Sep 23 22:19:24 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201964, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.129.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 22:19:25 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201964, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.129.84@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
Sep 23 22:19:25 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201964, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.129.84@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
Sep 23 22:19:27 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:19:27.850116 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35771
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:35771
--ranger_config_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:58735/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:27.945396 13738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:27.945614 13738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:27.945636 13738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:27.946842 13738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:27.946888 13738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:27.946902 13738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:27.946914 13738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:27.946926 13738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:27.948642 13738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.129.84:58735/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:35771
--ranger_config_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:27.948889 13738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:27.949131 13738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:27.954552 13744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:27.954586 13738 server_base.cc:1034] running on GCE node
W20260923 22:19:27.954607 13746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:27.954559 13743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:27.954957 13738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:27.955212 13738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:27.956377 13738 hybrid_clock.cc:648] HybridClock initialized: now 1790201967956361 us; error 28 us; skew 500 ppm
Sep 23 22:19:27 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201967, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:27.963245 13738 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:19:27.963706 13738 webserver.cc:460] Webserver started at http://127.1.129.126:35987/ using document root <none> and password file <none>
I20260923 22:19:27.963953 13738 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:27.964023 13738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:27.964180 13738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:19:27 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201967, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:28.018806 13738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "aea92274bca348ef80f65d76970a3af6"
format_stamp: "Formatted at 2026-09-23 22:19:28 on dist-test-slave-dmw7"
server_key: "3e505658c5641959c1fa1ef04f82b1d1"
server_key_iv: "87fee1d6abb7b2980e0bd843b68eac15"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:28.019285 13738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "aea92274bca348ef80f65d76970a3af6"
format_stamp: "Formatted at 2026-09-23 22:19:28 on dist-test-slave-dmw7"
server_key: "3e505658c5641959c1fa1ef04f82b1d1"
server_key_iv: "87fee1d6abb7b2980e0bd843b68eac15"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:28.021186 13738 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:28.037225 13753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:28.037503 13738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:28.037683 13738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "aea92274bca348ef80f65d76970a3af6"
format_stamp: "Formatted at 2026-09-23 22:19:28 on dist-test-slave-dmw7"
server_key: "3e505658c5641959c1fa1ef04f82b1d1"
server_key_iv: "87fee1d6abb7b2980e0bd843b68eac15"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:28.037853 13738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:28.051021 13738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:28.053049 13738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:28.053225 13738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:28.058234 13738 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:35771
I20260923 22:19:28.058316 13805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:35771 every 8 connection(s)
I20260923 22:19:28.058765 13738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:28.059458 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:28.061700 13806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Bootstrap starting.
I20260923 22:19:28.062314 13806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:28.062677 13806 log.cc:824] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:28.063426 13806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: No bootstrap required, opened a new log
I20260923 22:19:28.065475 13806 raft_consensus.cc:348] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } }
I20260923 22:19:28.065632 13806 raft_consensus.cc:374] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:28.065657 13806 raft_consensus.cc:729] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea92274bca348ef80f65d76970a3af6, State: Initialized, Role: FOLLOWER
I20260923 22:19:28.065817 13806 consensus_queue.cc:260] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } }
I20260923 22:19:28.065896 13806 raft_consensus.cc:388] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:28.065917 13806 raft_consensus.cc:482] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:28.065994 13806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:28.066546 13806 raft_consensus.cc:504] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } }
I20260923 22:19:28.066680 13806 leader_election.cc:302] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aea92274bca348ef80f65d76970a3af6; no voters:
I20260923 22:19:28.066876 13806 leader_election.cc:288] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:19:28.066902 13809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:28.067029 13809 raft_consensus.cc:686] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 1 LEADER]: Becoming Leader. State: Replica: aea92274bca348ef80f65d76970a3af6, State: Running, Role: LEADER
I20260923 22:19:28.067112 13809 consensus_queue.cc:237] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } }
I20260923 22:19:28.067189 13806 sys_catalog.cc:564] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:28.067417 13809 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea92274bca348ef80f65d76970a3af6. Latest consensus state: current_term: 1 leader_uuid: "aea92274bca348ef80f65d76970a3af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } } }
I20260923 22:19:28.067428 13810 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aea92274bca348ef80f65d76970a3af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } } }
I20260923 22:19:28.067502 13809 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:28.067548 13810 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:28.067947 13813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:28.068536 13813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:28.068907 13806 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 22:19:28.068967 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 13738
I20260923 22:19:28.069061 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:19:28.069234 1541 external_mini_cluster.cc:1428] Setting key 147a7c72ef4e3373ebd034da65a89bfb
I20260923 22:19:28.075377 13813 catalog_manager.cc:1347] Generated new cluster ID: f60a6db269364dddbcddc088d8827d33
I20260923 22:19:28.075472 13813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:19:28 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:28.092703 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:28.071308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.1.129.126:35771)
0923 22:19:28.074790 (+ 3482us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:28.074807 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:28.074850 (+ 43us) server_negotiation.cc:407] Connection header received
0923 22:19:28.075075 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:28.075083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:28.075158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:28.081223 (+ 6065us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:28.081447 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:28.083008 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:28.084189 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:28.084920 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:28.085031 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:28.088297 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:28.088303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:28.088304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:28.088319 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:28.090516 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:28.090966 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:28.090969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:28.090970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:28.091112 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:28.091313 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:28.091314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:28.091314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:28.091648 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:28.091800 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:28.092326 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 22:19:28.092460 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3424,"thread_start_us":424,"threads_started":1}
I20260923 22:19:28.099628 13813 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:28.100363 13813 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:28.111670 13813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Generated new TSK 0
I20260923 22:19:28.111899 13813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:19:28.826041 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:58735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35771
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:28.944450 13851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:28.944684 13851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:28.944711 13851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:28.946069 13851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:28.946135 13851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:28.946182 13851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:19:28.947801 13851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.129.84:58735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:28.948086 13851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:28.948335 13851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:28.954744 13857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:28.954843 13851 server_base.cc:1034] running on GCE node
W20260923 22:19:28.954942 13859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:28.954738 13856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:28.955233 13851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:28.955508 13851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:28.956674 13851 hybrid_clock.cc:648] HybridClock initialized: now 1790201968956660 us; error 31 us; skew 500 ppm
Sep 23 22:19:28 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201968, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:28.959455 13851 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:19:28.960005 13851 webserver.cc:460] Webserver started at http://127.1.129.65:45387/ using document root <none> and password file <none>
I20260923 22:19:28.960278 13851 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:28.960348 13851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:28.960480 13851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:19:28 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201968, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:28.977586 13851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "b06422fa36984b79a662e34e6c760e64"
format_stamp: "Formatted at 2026-09-23 22:19:28 on dist-test-slave-dmw7"
server_key: "951b45e710e999b77dcc788de3cf690f"
server_key_iv: "f9dfd58e80e046fa8e3107835cb3ecdc"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:28.978113 13851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "b06422fa36984b79a662e34e6c760e64"
format_stamp: "Formatted at 2026-09-23 22:19:28 on dist-test-slave-dmw7"
server_key: "951b45e710e999b77dcc788de3cf690f"
server_key_iv: "f9dfd58e80e046fa8e3107835cb3ecdc"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:28.982120 13851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260923 22:19:29.000118 13868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.000408 13851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:29.000586 13851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "b06422fa36984b79a662e34e6c760e64"
format_stamp: "Formatted at 2026-09-23 22:19:28 on dist-test-slave-dmw7"
server_key: "951b45e710e999b77dcc788de3cf690f"
server_key_iv: "f9dfd58e80e046fa8e3107835cb3ecdc"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.000775 13851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:29.034933 13851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:29.037752 13851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:29.037968 13851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:29.038247 13851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:29.038709 13851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:29.038811 13851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.038908 13851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:29.038985 13851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.046447 13851 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:36705
I20260923 22:19:29.046525 13981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:36705 every 8 connection(s)
I20260923 22:19:29.046989 13851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:29.053216 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 13851
I20260923 22:19:29.053326 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:19:29.053494 1541 external_mini_cluster.cc:1428] Setting key bf316fcd3ac3b39d57e652a7c9e54325
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201968, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:29.074362 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:29.048095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35771 (local address 127.1.129.65:54683)
0923 22:19:29.048326 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:29.048383 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:29.049049 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:29.054137 (+ 5088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:29.054155 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:29.054438 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:29.054767 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:29.054774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.055797 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.055798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:29.057113 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:29.057116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.057615 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.057616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:29.057749 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:29.058371 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:29.058381 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:29.060110 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:29.062658 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:29.062665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:29.062668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:29.062977 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:29.073110 (+ 10133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:29.073112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:29.073112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:29.073303 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:29.073636 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:29.073645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:29.073831 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 22:19:29.073923 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 22:19:29.074105 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
I20260923 22:19:29.074908 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:29.048394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:54683 (local address 127.1.129.126:35771)
0923 22:19:29.053877 (+ 5483us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:29.053879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:29.053891 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:19:29.053917 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:29.053918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:29.053930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:29.054867 (+ 937us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:29.054907 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.055678 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.057228 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.057513 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.060536 (+ 3023us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:29.060592 (+ 56us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:29.060593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:29.060594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:29.060599 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:29.062514 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:29.072872 (+ 10358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:29.072875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:29.072875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:29.072985 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:29.073383 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:29.073384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:29.073384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:29.073539 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:29.074674 (+ 1135us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:29.074718 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 22:19:29.074813 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2351,"thread_start_us":84,"threads_started":1}
I20260923 22:19:29.075364 13982 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35771
I20260923 22:19:29.075461 13982 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:29.075649 13982 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:29.076159 13770 ts_manager.cc:194] Registered new tserver with Master: b06422fa36984b79a662e34e6c760e64 (127.1.129.65:36705)
I20260923 22:19:29.077025 13770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:54683
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:19:29.111735 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:58735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35771
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 22:19:29.253160 13990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:29.253432 13990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:29.253468 13990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:29.255230 13990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:29.255292 13990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:29.255331 13990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:19:29.257493 13990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.129.84:58735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:29.257786 13990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:29.258066 13990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:29.264717 13996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:29.264775 13998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:29.264837 13990 server_base.cc:1034] running on GCE node
W20260923 22:19:29.264721 13995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:29.265149 13990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:29.265424 13990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:29.272742 13990 hybrid_clock.cc:648] HybridClock initialized: now 1790201969272717 us; error 40 us; skew 500 ppm
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:29.275460 13990 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:19:29.276005 13990 webserver.cc:460] Webserver started at http://127.1.129.66:43183/ using document root <none> and password file <none>
I20260923 22:19:29.276243 13990 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:29.276309 13990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:29.276460 13990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:29.290316 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "b5732dc603724d999151a91fb9c3217c"
format_stamp: "Formatted at 2026-09-23 22:19:29 on dist-test-slave-dmw7"
server_key: "d345581c1eeabb8d78ccc4551825f830"
server_key_iv: "02d966d1fd5dc479c7fb540fb9f039f5"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.290812 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "b5732dc603724d999151a91fb9c3217c"
format_stamp: "Formatted at 2026-09-23 22:19:29 on dist-test-slave-dmw7"
server_key: "d345581c1eeabb8d78ccc4551825f830"
server_key_iv: "02d966d1fd5dc479c7fb540fb9f039f5"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.292735 13990 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260923 22:19:29.305910 14006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.306131 13990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:29.306231 13990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "b5732dc603724d999151a91fb9c3217c"
format_stamp: "Formatted at 2026-09-23 22:19:29 on dist-test-slave-dmw7"
server_key: "d345581c1eeabb8d78ccc4551825f830"
server_key_iv: "02d966d1fd5dc479c7fb540fb9f039f5"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.306313 13990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:29.345932 13990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:29.348114 13990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:29.348297 13990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:29.348539 13990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:29.348919 13990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:29.348969 13990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.349009 13990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:29.349035 13990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.356396 13990 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:45415
I20260923 22:19:29.356765 13990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:19:29.357270 14119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:45415 every 8 connection(s)
I20260923 22:19:29.361655 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 13990
I20260923 22:19:29.361756 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:19:29.361953 1541 external_mini_cluster.cc:1428] Setting key f96f723634c091a752e6ee7f320fd21a
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:29.366658 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:29.357887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:37211 (local address 127.1.129.126:35771)
0923 22:19:29.357932 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:29.357934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:29.357945 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:19:29.358387 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:29.358389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:29.358405 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:29.358458 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:29.359108 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.359627 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.360648 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.360868 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.360947 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:29.363445 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:29.363447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:29.363448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:29.363456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:29.365052 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:29.365466 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:29.365467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:29.365468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:29.365528 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:29.365831 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:29.365832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:29.365832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:29.365979 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:29.366021 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:29.366400 (+ 379us) server_negotiation.cc:299] Negotiation successful
0923 22:19:29.366502 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:19:29.367278 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:29.357518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35771 (local address 127.1.129.66:37211)
0923 22:19:29.357706 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:29.357750 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:29.358282 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:29.358538 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:29.358542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:29.358815 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:29.359012 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:29.359017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.359718 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.359720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:29.360552 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:29.360554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.360967 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.360968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:29.361095 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:29.361673 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:29.361683 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:29.363336 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:29.365147 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:29.365155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:29.365157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:29.365396 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:29.365605 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:29.365605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:29.365606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:29.365723 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:29.366067 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:29.366084 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:29.366252 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 22:19:29.366802 (+ 550us) client_negotiation.cc:240] Negotiation successful
0923 22:19:29.366979 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
I20260923 22:19:29.367719 14120 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35771
I20260923 22:19:29.367806 14120 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:29.367997 14120 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:29.368361 13770 ts_manager.cc:194] Registered new tserver with Master: b5732dc603724d999151a91fb9c3217c (127.1.129.66:45415)
I20260923 22:19:29.368882 13770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:37211
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:19:29.389531 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:58735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35771
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:29.484458 14125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:29.484686 14125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:29.484726 14125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:29.486403 14125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:29.486469 14125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:29.486509 14125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:19:29.488634 14125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.129.84:58735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:29.488874 14125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:29.489164 14125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:29.495533 14131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:29.495829 14133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:29.495679 14130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:29.496206 14125 server_base.cc:1034] running on GCE node
I20260923 22:19:29.496436 14125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:29.496703 14125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:29.497866 14125 hybrid_clock.cc:648] HybridClock initialized: now 1790201969497845 us; error 35 us; skew 500 ppm
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:29.500540 14125 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:19:29.500973 14125 webserver.cc:460] Webserver started at http://127.1.129.67:44163/ using document root <none> and password file <none>
I20260923 22:19:29.501223 14125 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:29.501287 14125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:29.501426 14125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:29.513322 14125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "0b3b0b551f634cd6922c9ba1092a8f67"
format_stamp: "Formatted at 2026-09-23 22:19:29 on dist-test-slave-dmw7"
server_key: "738b2fb6b55edb74856281a8072a6aa2"
server_key_iv: "f756c89e1b29cc788061961b296cb294"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.513777 14125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "0b3b0b551f634cd6922c9ba1092a8f67"
format_stamp: "Formatted at 2026-09-23 22:19:29 on dist-test-slave-dmw7"
server_key: "738b2fb6b55edb74856281a8072a6aa2"
server_key_iv: "f756c89e1b29cc788061961b296cb294"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.520606 14125 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260923 22:19:29.533663 14140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.533847 14125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:29.533937 14125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "0b3b0b551f634cd6922c9ba1092a8f67"
format_stamp: "Formatted at 2026-09-23 22:19:29 on dist-test-slave-dmw7"
server_key: "738b2fb6b55edb74856281a8072a6aa2"
server_key_iv: "f756c89e1b29cc788061961b296cb294"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.534039 14125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:29.544458 14125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:29.546952 14125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:29.547120 14125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:29.547391 14125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:29.547765 14125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:29.547811 14125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.547844 14125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:29.547869 14125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.555050 14125 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:35007
I20260923 22:19:29.555649 14125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:19:29.558844 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 14125
I20260923 22:19:29.558990 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:19:29.559216 1541 external_mini_cluster.cc:1428] Setting key 59a1059c9f74f15eaf48ab822d004088
I20260923 22:19:29.567312 14253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:35007 every 8 connection(s)
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:29.579545 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:29.568649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:46713 (local address 127.1.129.126:35771)
0923 22:19:29.568690 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:29.568692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:29.568703 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:19:29.569350 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:29.569351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:29.569365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:29.569428 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:29.570217 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.571273 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.573061 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.573364 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.573408 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:29.576132 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:29.576134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:29.576134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:29.576141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:29.577858 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:29.578360 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:29.578361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:29.578362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:29.578423 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:29.578773 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:29.578784 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:29.578785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:29.578925 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:29.578970 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:29.579346 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 22:19:29.579433 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:19:29.580286 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:29.568335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35771 (local address 127.1.129.67:46713)
0923 22:19:29.568553 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:29.568607 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:29.569223 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:29.569525 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:29.569529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:29.569794 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:29.570080 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:29.570087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.571611 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.571612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:29.572957 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:29.572961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.573527 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.573528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:29.573665 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:29.574276 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:29.574286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:29.576022 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:29.577954 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:29.577961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:29.577964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:29.578265 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:29.578510 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:29.578510 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:29.578511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:29.578686 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:29.579017 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:29.579023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:29.579190 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 22:19:29.579871 (+ 681us) client_negotiation.cc:240] Negotiation successful
0923 22:19:29.580045 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260923 22:19:29.581041 14254 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35771
I20260923 22:19:29.581135 14254 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:29.581315 14254 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:29.581765 13770 ts_manager.cc:194] Registered new tserver with Master: 0b3b0b551f634cd6922c9ba1092a8f67 (127.1.129.67:35007)
I20260923 22:19:29.582286 13770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:46713
I20260923 22:19:29.583617 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:29.584278 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 13851
I20260923 22:19:29.592139 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 13990
I20260923 22:19:29.598694 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 14125
I20260923 22:19:29.604028 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 13738
I20260923 22:19:29.609588 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35771
--webserver_interface=127.1.129.126
--webserver_port=35987
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:35771
--ranger_config_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:58735/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.129.84:58735/kms with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:29.690752 14257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:29.690971 14257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:29.691002 14257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:29.692214 14257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:29.692260 14257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:29.692276 14257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:29.692287 14257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:29.692339 14257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:29.693997 14257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.129.84:58735/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:35771
--ranger_config_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=35987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:29.694235 14257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:29.694461 14257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:29.699363 14265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:29.699472 14262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:29.699611 14263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:29.699615 14257 server_base.cc:1034] running on GCE node
I20260923 22:19:29.700068 14257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:29.700410 14257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:29.701563 14257 hybrid_clock.cc:648] HybridClock initialized: now 1790201969701548 us; error 31 us; skew 500 ppm
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:29.704043 14257 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:19:29.704473 14257 webserver.cc:460] Webserver started at http://127.1.129.126:35987/ using document root <none> and password file <none>
I20260923 22:19:29.704710 14257 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:29.704782 14257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:29.766312 14257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:29.767163 14273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.767365 14257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:29.767495 14257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "aea92274bca348ef80f65d76970a3af6"
format_stamp: "Formatted at 2026-09-23 22:19:28 on dist-test-slave-dmw7"
server_key: "3e505658c5641959c1fa1ef04f82b1d1"
server_key_iv: "87fee1d6abb7b2980e0bd843b68eac15"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.767990 14257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:29.788262 14257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:29.789913 14257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:29.790040 14257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:29.794564 14257 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:35771
I20260923 22:19:29.794627 14325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:35771 every 8 connection(s)
I20260923 22:19:29.795058 14257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:29.795223 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 14257
I20260923 22:19:29.795542 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:58735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:36705
--local_ip_for_outbound_sockets=127.1.129.65
--tserver_master_addrs=127.1.129.126:35771
--webserver_port=45387
--webserver_interface=127.1.129.65
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.129.84:58735/kms with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:29.796458 14326 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:19:29.797150 14326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Bootstrap starting.
I20260923 22:19:29.805531 14326 log.cc:824] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:29.807380 14326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:29.807684 14326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Bootstrap complete.
I20260923 22:19:29.810330 14326 raft_consensus.cc:348] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } }
I20260923 22:19:29.810609 14326 raft_consensus.cc:729] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea92274bca348ef80f65d76970a3af6, State: Initialized, Role: FOLLOWER
I20260923 22:19:29.810827 14326 consensus_queue.cc:260] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } }
I20260923 22:19:29.810925 14326 raft_consensus.cc:388] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:29.810997 14326 raft_consensus.cc:482] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:29.811053 14326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:19:29.812108 14326 raft_consensus.cc:504] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } }
I20260923 22:19:29.812233 14326 leader_election.cc:302] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aea92274bca348ef80f65d76970a3af6; no voters:
I20260923 22:19:29.812434 14326 leader_election.cc:288] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 22:19:29.812484 14330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:19:29.812610 14330 raft_consensus.cc:686] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [term 2 LEADER]: Becoming Leader. State: Replica: aea92274bca348ef80f65d76970a3af6, State: Running, Role: LEADER
I20260923 22:19:29.812719 14326 sys_catalog.cc:564] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:29.812712 14330 consensus_queue.cc:237] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } }
I20260923 22:19:29.813225 14331 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aea92274bca348ef80f65d76970a3af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } } }
I20260923 22:19:29.813323 14331 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:29.813225 14332 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea92274bca348ef80f65d76970a3af6. Latest consensus state: current_term: 2 leader_uuid: "aea92274bca348ef80f65d76970a3af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea92274bca348ef80f65d76970a3af6" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35771 } } }
I20260923 22:19:29.813512 14332 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:29.813627 14335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:29.814121 14335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:29.814235 14335 catalog_manager.cc:1259] Loaded cluster ID: f60a6db269364dddbcddc088d8827d33
I20260923 22:19:29.814270 14335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:19:29.814596 14326 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 22:19:29.815536 14335 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:29.815726 14335 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Loaded TSK: 0
I20260923 22:19:29.815970 14335 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:19:29.885950 14328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:29.886178 14328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:29.886219 14328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:29.887413 14328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:29.887475 14328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:29.887518 14328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:19:29.888993 14328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.129.84:58735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:36705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=45387
--tserver_master_addrs=127.1.129.126:35771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:29.889263 14328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:29.889489 14328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:29.895835 14361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:29.895835 14362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:29.896265 14364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:29.897521 14328 server_base.cc:1034] running on GCE node
I20260923 22:19:29.897801 14328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:29.898066 14328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:29.905431 14328 hybrid_clock.cc:648] HybridClock initialized: now 1790201969905409 us; error 40 us; skew 500 ppm
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:29.908346 14328 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:19:29.908908 14328 webserver.cc:460] Webserver started at http://127.1.129.65:45387/ using document root <none> and password file <none>
I20260923 22:19:29.909157 14328 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:29.909236 14328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:29.933311 14328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:29.934090 14374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.934290 14328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:29.934376 14328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "b06422fa36984b79a662e34e6c760e64"
format_stamp: "Formatted at 2026-09-23 22:19:28 on dist-test-slave-dmw7"
server_key: "951b45e710e999b77dcc788de3cf690f"
server_key_iv: "f9dfd58e80e046fa8e3107835cb3ecdc"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:29.935318 14328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:29.965113 14328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:29.967633 14328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:29.967813 14328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:29.968068 14328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:29.968446 14328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:29.968497 14328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.968535 14328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:29.968560 14328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:29.975466 14328 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:36705
I20260923 22:19:29.975926 14328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:29.979005 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:36705 every 8 connection(s)
I20260923 22:19:29.985014 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 14328
I20260923 22:19:29.985318 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:58735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:45415
--local_ip_for_outbound_sockets=127.1.129.66
--tserver_master_addrs=127.1.129.126:35771
--webserver_port=43183
--webserver_interface=127.1.129.66
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.129.84:58735/kms with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:19:29 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:29.992610 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:29.978007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35771 (local address 127.1.129.65:44151)
0923 22:19:29.979185 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:29.979226 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:29.979825 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:29.980631 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:29.980635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:29.980903 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:29.981207 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:29.981214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.982623 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.982625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:29.983630 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:29.983632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.984385 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.984386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:29.984508 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:29.986853 (+ 2345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:29.986870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:29.988500 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:29.990579 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:29.990586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:29.990588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:29.990943 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:29.991256 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:29.991257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:29.991258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:29.991394 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:29.991874 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:29.991880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:29.992039 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 22:19:29.992148 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 22:19:29.992314 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"thread_start_us":110,"threads_started":1}
I20260923 22:19:29.993335 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:29.978430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:44151 (local address 127.1.129.126:35771)
0923 22:19:29.980042 (+ 1612us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:29.980051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:29.980101 (+ 50us) server_negotiation.cc:407] Connection header received
0923 22:19:29.980241 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:29.980247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:29.980302 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:29.980439 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:29.981343 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.982510 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.983730 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:29.984280 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:29.984399 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:29.988640 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:29.988644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:29.988644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:29.988656 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:29.990462 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:29.991042 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:29.991045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:29.991045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:29.991178 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:29.991464 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:29.991465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:29.991465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:29.991769 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:29.991846 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:29.992952 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0923 22:19:29.993092 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1555,"thread_start_us":88,"threads_started":1}
I20260923 22:19:29.994027 14488 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35771
I20260923 22:19:29.994148 14488 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:29.994365 14488 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
W20260923 22:19:29.995307 14488 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:35771 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 22:19:29.995388 14488 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:29.995456 14488 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:29.995771 14488 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:29.995850 14488 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
W20260923 22:19:29.996165 14488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:19:30.098111 14492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:30.098459 14492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:30.098526 14492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:30.100389 14492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:30.100467 14492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:30.100513 14492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:19:30.102195 14492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.129.84:58735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:45415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=43183
--tserver_master_addrs=127.1.129.126:35771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:30.102487 14492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:30.102818 14492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:30.111701 14497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:30.111722 14500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:30.112041 14492 server_base.cc:1034] running on GCE node
W20260923 22:19:30.112960 14498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:30.113240 14492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:30.113513 14492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:30.114707 14492 hybrid_clock.cc:648] HybridClock initialized: now 1790201970114691 us; error 33 us; skew 500 ppm
Sep 23 22:19:30 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:30.117940 14492 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:19:30.118474 14492 webserver.cc:460] Webserver started at http://127.1.129.66:43183/ using document root <none> and password file <none>
I20260923 22:19:30.118755 14492 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:30.118820 14492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 22:19:30 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:30.138913 14492 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260923 22:19:30.140148 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:30.140377 14492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:30.140482 14492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "b5732dc603724d999151a91fb9c3217c"
format_stamp: "Formatted at 2026-09-23 22:19:29 on dist-test-slave-dmw7"
server_key: "d345581c1eeabb8d78ccc4551825f830"
server_key_iv: "02d966d1fd5dc479c7fb540fb9f039f5"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:30.140969 14492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:30.161316 14492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:30.163544 14492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:30.163715 14492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:30.163982 14492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:30.164355 14492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:30.164391 14492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:30.164430 14492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:30.164458 14492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:30.171692 14492 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:45415
I20260923 22:19:30.172230 14492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:19:30.172750 14621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:45415 every 8 connection(s)
I20260923 22:19:30.177807 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 14492
I20260923 22:19:30.178131 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.129.84:58735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:35007
--local_ip_for_outbound_sockets=127.1.129.67
--tserver_master_addrs=127.1.129.126:35771
--webserver_port=44163
--webserver_interface=127.1.129.67
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.129.84:58735/kms with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:19:30 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:30.183670 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:30.172949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:56313 (local address 127.1.129.126:35771)
0923 22:19:30.172998 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:30.173001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:30.173257 (+ 256us) server_negotiation.cc:407] Connection header received
0923 22:19:30.173937 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:30.173938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:30.173954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:30.173985 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:30.174859 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.175626 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.177161 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.177451 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.177499 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:30.180093 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:30.180095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:30.180095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:30.180103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:30.181865 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:30.182425 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:30.182426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:30.182426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:30.182503 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:30.182844 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:30.182845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:30.182845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:30.183002 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:30.183086 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:30.183453 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 22:19:30.183568 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:19:30.184160 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:30.172974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35771 (local address 127.1.129.66:56313)
0923 22:19:30.173122 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:30.173168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:30.173813 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:30.174163 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:30.174167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:30.174389 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:30.174742 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:30.174749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.175749 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.175750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:30.177050 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:30.177054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.177632 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.177633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:30.177777 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:30.178406 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:30.178415 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:30.179968 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:30.181968 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:30.181977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:30.181980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:30.182326 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:30.182591 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:30.182593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:30.182593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:30.182770 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:30.183091 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:30.183096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:30.183280 (+ 184us) client_negotiation.cc:769] Sending connection context
0923 22:19:30.183387 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 22:19:30.183559 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260923 22:19:30.184669 14622 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35771
I20260923 22:19:30.184783 14622 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:30.184998 14622 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
W20260923 22:19:30.185864 14622 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:35771 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 22:19:30.185947 14622 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:30.186025 14622 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:30.186467 14622 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:30.186553 14622 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
W20260923 22:19:30.186960 14622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:19:30.280318 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:30.280602 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:30.280637 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:30.282342 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:30.282403 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:30.282444 14625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:19:30.284801 14625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.129.84:58735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:35007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=44163
--tserver_master_addrs=127.1.129.126:35771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:30.285061 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:30.285357 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:30.291877 14630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:30.291915 14631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:30.291896 14633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:30.292714 14625 server_base.cc:1034] running on GCE node
I20260923 22:19:30.292999 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:30.293291 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:30.294472 14625 hybrid_clock.cc:648] HybridClock initialized: now 1790201970294446 us; error 41 us; skew 500 ppm
Sep 23 22:19:30 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:30.297794 14625 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:19:30.298369 14625 webserver.cc:460] Webserver started at http://127.1.129.67:44163/ using document root <none> and password file <none>
I20260923 22:19:30.298712 14625 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:30.298830 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 22:19:30 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:30.327541 14625 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260923 22:19:30.329183 14642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:30.329366 14625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:30.329458 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "0b3b0b551f634cd6922c9ba1092a8f67"
format_stamp: "Formatted at 2026-09-23 22:19:29 on dist-test-slave-dmw7"
server_key: "738b2fb6b55edb74856281a8072a6aa2"
server_key_iv: "f756c89e1b29cc788061961b296cb294"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:30.329969 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:30.346606 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:30.349125 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:30.349303 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:30.349591 14625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:30.349980 14625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:30.350032 14625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:30.350075 14625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:30.350093 14625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:30.358316 14625 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:35007
I20260923 22:19:30.358402 14755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:35007 every 8 connection(s)
I20260923 22:19:30.358857 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:19:30 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:30.366458 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 14625
I20260923 22:19:30.368618 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:30.359634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:45963 (local address 127.1.129.126:35771)
0923 22:19:30.359705 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:30.359708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:30.359944 (+ 236us) server_negotiation.cc:407] Connection header received
0923 22:19:30.360479 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:30.360480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:30.360492 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:30.360543 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:30.361159 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.361660 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.362775 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.362979 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.363060 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:30.365303 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:30.365305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:30.365305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:30.365318 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:30.366936 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:30.367459 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:30.367460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:30.367460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:30.367527 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:30.367839 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:30.367840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:30.367840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:30.367974 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:30.368041 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:30.368404 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 22:19:30.368505 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:19:30.368757 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:30.359629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35771 (local address 127.1.129.67:45963)
0923 22:19:30.359803 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:30.359846 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:30.360326 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:30.360628 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:30.360633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:30.360824 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:30.361066 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:30.361077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.361756 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.361757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:30.362670 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:30.362673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.363084 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.363085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:30.363193 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:30.363712 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:30.363721 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:30.365188 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:30.367031 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:30.367041 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:30.367044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:30.367373 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:30.367623 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:30.367624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:30.367624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:30.367774 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:30.368066 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:30.368072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:30.368241 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 22:19:30.368354 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 22:19:30.368515 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260923 22:19:30.369386 14756 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35771
I20260923 22:19:30.369488 14756 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:30.369673 14756 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
W20260923 22:19:30.370361 14756 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:35771 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 22:19:30.370442 14756 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:30.370508 14756 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:30.370877 14756 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:30.370959 14756 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
W20260923 22:19:30.371250 14756 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:19:30.375181 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:30.366962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.1.129.126:35771)
0923 22:19:30.367680 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:30.367682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:30.367689 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:30.367708 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:30.367709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:30.367721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:30.367777 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:30.368565 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.369341 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.370379 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:30.370719 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:30.370779 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:30.372153 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:30.372155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:30.372156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:30.372163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:30.373883 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:30.374238 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:30.374239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:30.374240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:30.374305 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:30.374571 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:30.374572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:30.374573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:30.374746 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:30.374933 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:30.374971 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 22:19:30.375069 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":100,"threads_started":1}
Sep 23 22:19:30 dist-test-slave-dmw7 krb5kdc[13294](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:30.996356 14488 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:30.996531 14488 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:30.996950 14290 ts_manager.cc:194] Registered new tserver with Master: b06422fa36984b79a662e34e6c760e64 (127.1.129.65:36705)
I20260923 22:19:30.997633 14290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:44151
I20260923 22:19:31.187143 14622 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:31.187296 14622 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:31.187629 14290 ts_manager.cc:194] Registered new tserver with Master: b5732dc603724d999151a91fb9c3217c (127.1.129.66:45415)
I20260923 22:19:31.188161 14290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:56313
Sep 23 22:19:31 dist-test-slave-dmw7 krb5kdc[13294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for HTTP/127.1.129.84@KRBTEST.COM
I20260923 22:19:31.371477 14756 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:31.371675 14756 heartbeater.cc:511] Master 127.1.129.126:35771 requested a full tablet report, sending...
I20260923 22:19:31.372066 14290 ts_manager.cc:194] Registered new tserver with Master: 0b3b0b551f634cd6922c9ba1092a8f67 (127.1.129.67:35007)
I20260923 22:19:31.372462 14290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:45963
I20260923 22:19:31.377683 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:31.385633 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.378674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.1.129.126:35771)
0923 22:19:31.378781 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:31.378783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:31.378792 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:31.378840 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:31.378840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:31.378851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:31.378909 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:31.379591 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.380113 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.381142 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.381327 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.381375 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.383013 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:31.383014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:31.383014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:31.383020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:31.384495 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:31.384931 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:31.384931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:31.384932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:31.384981 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:31.385174 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:31.385175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:31.385175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:31.385271 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:31.385359 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:31.385496 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 22:19:31.385562 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260923 22:19:31.387701 14290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:19:31.389717 14290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:19:31.403257 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.396607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:35007 (local address 127.0.0.1:58566)
0923 22:19:31.398760 (+ 2153us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:31.398772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:31.398824 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:31.399294 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:31.399298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:31.399300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:31.399373 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.399375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.400588 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.400589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.401863 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.401866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.402958 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.402959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.403031 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.403045 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 22:19:31.403119 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:19:31.403170 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2133,"mutex_wait_us":41,"thread_start_us":74,"threads_started":1}
I20260923 22:19:31.403517 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.396350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:45415 (local address 127.0.0.1:53180)
0923 22:19:31.396910 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:31.397078 (+ 168us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:31.397154 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:31.399385 (+ 2231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:31.399386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:31.399388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:31.399441 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.399442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.400891 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.400892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.402295 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.402298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.403301 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.403302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.403364 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.403370 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:19:31.403404 (+ 34us) client_negotiation.cc:240] Negotiation successful
0923 22:19:31.403446 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":86,"threads_started":1}
I20260923 22:19:31.403582 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.396692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.1.129.67:35007)
0923 22:19:31.398816 (+ 2124us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:31.398821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:31.398835 (+ 14us) server_negotiation.cc:407] Connection header received
0923 22:19:31.399176 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:31.399179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:31.399195 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:31.399271 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:31.399462 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.400491 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.401960 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.402803 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.402868 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.402926 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:31.403469 (+ 543us) server_negotiation.cc:299] Negotiation successful
0923 22:19:31.403513 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2084,"thread_start_us":53,"threads_started":1}
I20260923 22:19:31.404217 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.396453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.1.129.66:45415)
0923 22:19:31.398973 (+ 2520us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:31.398977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:31.398986 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:31.399021 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:31.399023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:31.399039 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:31.399516 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:31.399592 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.400646 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.402400 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.403200 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.403697 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.403769 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:31.403793 (+ 24us) server_negotiation.cc:299] Negotiation successful
0923 22:19:31.403832 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2486,"thread_start_us":63,"threads_started":1}
I20260923 22:19:31.404241 14689 tablet_service.cc:1467] Processing CreateTablet for tablet 265fcb83256e46a8a51f01ef133ab3ef (DEFAULT_TABLE table=test-table [id=c9c9e650d6134e0aa98cc84255e6d7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:31.404606 14689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265fcb83256e46a8a51f01ef133ab3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:31.407263 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 265fcb83256e46a8a51f01ef133ab3ef (DEFAULT_TABLE table=test-table [id=c9c9e650d6134e0aa98cc84255e6d7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:31.407629 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265fcb83256e46a8a51f01ef133ab3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:31.408658 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.396785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:36705 (local address 127.0.0.1:56132)
0923 22:19:31.400760 (+ 3975us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:31.400770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:31.400810 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:31.404563 (+ 3753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:31.404565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:31.404567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:31.404608 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.404609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.405992 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.405993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.407118 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.407121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.408430 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.408431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.408492 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.408498 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:19:31.408530 (+ 32us) client_negotiation.cc:240] Negotiation successful
0923 22:19:31.408572 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3958,"thread_start_us":60,"threads_started":1}
I20260923 22:19:31.408804 14789 tablet_bootstrap.cc:492] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: Bootstrap starting.
I20260923 22:19:31.409051 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.396886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.1.129.65:36705)
0923 22:19:31.404381 (+ 7495us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:31.404387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:31.404399 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:19:31.404436 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:31.404438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:31.404455 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:31.404682 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:31.404762 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.405887 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.407229 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.408328 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.408839 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.408909 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:31.408935 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 22:19:31.408974 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7455,"thread_start_us":61,"threads_started":1}
I20260923 22:19:31.409724 14422 tablet_service.cc:1467] Processing CreateTablet for tablet 265fcb83256e46a8a51f01ef133ab3ef (DEFAULT_TABLE table=test-table [id=c9c9e650d6134e0aa98cc84255e6d7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:31.410063 14422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265fcb83256e46a8a51f01ef133ab3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:31.410149 14789 tablet_bootstrap.cc:654] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:31.410482 14789 log.cc:824] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:31.411280 14789 tablet_bootstrap.cc:492] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: No bootstrap required, opened a new log
I20260923 22:19:31.411356 14789 ts_tablet_manager.cc:1397] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 22:19:31.413755 14789 raft_consensus.cc:348] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.413925 14789 raft_consensus.cc:374] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:31.413960 14789 raft_consensus.cc:729] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3b0b551f634cd6922c9ba1092a8f67, State: Initialized, Role: FOLLOWER
I20260923 22:19:31.414119 14789 consensus_queue.cc:260] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.414495 14789 ts_tablet_manager.cc:1428] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:19:31.414618 14756 heartbeater.cc:503] Master 127.1.129.126:35771 was elected leader, sending a full tablet report...
I20260923 22:19:31.414814 14791 tablet_bootstrap.cc:492] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Bootstrap starting.
I20260923 22:19:31.415495 14791 tablet_bootstrap.cc:654] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:31.415885 14791 log.cc:824] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:31.416733 14791 tablet_bootstrap.cc:492] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: No bootstrap required, opened a new log
I20260923 22:19:31.416810 14791 ts_tablet_manager.cc:1397] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:31.418814 14792 tablet_bootstrap.cc:492] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64: Bootstrap starting.
I20260923 22:19:31.419625 14792 tablet_bootstrap.cc:654] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:31.419559 14791 raft_consensus.cc:348] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.420125 14791 raft_consensus.cc:374] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:31.420189 14791 raft_consensus.cc:729] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5732dc603724d999151a91fb9c3217c, State: Initialized, Role: FOLLOWER
I20260923 22:19:31.420322 14791 consensus_queue.cc:260] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.420593 14791 ts_tablet_manager.cc:1428] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 22:19:31.420586 14792 log.cc:824] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:31.420744 14622 heartbeater.cc:503] Master 127.1.129.126:35771 was elected leader, sending a full tablet report...
I20260923 22:19:31.423281 14792 tablet_bootstrap.cc:492] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64: No bootstrap required, opened a new log
I20260923 22:19:31.423354 14792 ts_tablet_manager.cc:1397] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
W20260923 22:19:31.423655 14623 tablet.cc:2369] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:31.425910 14792 raft_consensus.cc:348] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.426095 14792 raft_consensus.cc:374] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:31.426144 14792 raft_consensus.cc:729] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06422fa36984b79a662e34e6c760e64, State: Initialized, Role: FOLLOWER
I20260923 22:19:31.426260 14792 consensus_queue.cc:260] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.426539 14792 ts_tablet_manager.cc:1428] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 22:19:31.426571 14488 heartbeater.cc:503] Master 127.1.129.126:35771 was elected leader, sending a full tablet report...
I20260923 22:19:31.442149 14795 raft_consensus.cc:482] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:31.442371 14795 raft_consensus.cc:504] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.443151 14795 leader_election.cc:288] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b3b0b551f634cd6922c9ba1092a8f67 (127.1.129.67:35007), b06422fa36984b79a662e34e6c760e64 (127.1.129.65:36705)
I20260923 22:19:31.446895 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.443282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:35351 (local address 127.1.129.65:36705)
0923 22:19:31.443327 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:31.443329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:31.443397 (+ 68us) server_negotiation.cc:407] Connection header received
0923 22:19:31.443546 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:31.443547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:31.443556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:31.443585 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:31.443797 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.444516 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.445930 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.446457 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.446501 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.446509 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:31.446784 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 22:19:31.446827 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:19:31.447140 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.443386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:57263 (local address 127.1.129.67:35007)
0923 22:19:31.443440 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:31.443442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:31.443506 (+ 64us) server_negotiation.cc:407] Connection header received
0923 22:19:31.443620 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:31.443621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:31.443631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:31.443667 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:31.443918 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.444633 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.446116 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.446742 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.446792 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.446801 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:31.447030 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 22:19:31.447073 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:19:31.447433 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.443046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:35007 (local address 127.1.129.66:57263)
0923 22:19:31.443445 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:31.443455 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:31.443531 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:31.443733 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:31.443734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:31.443736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:31.443806 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.443807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.444743 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.444744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.446012 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.446015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.446871 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.446871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.446951 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.446954 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:19:31.447258 (+ 304us) client_negotiation.cc:240] Negotiation successful
0923 22:19:31.447300 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":16,"thread_start_us":90,"threads_started":1}
I20260923 22:19:31.447436 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.442996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:36705 (local address 127.1.129.66:35351)
0923 22:19:31.443332 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:31.443344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:31.443468 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:31.443650 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:31.443651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:31.443652 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:31.443729 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.443731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.444611 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.444612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.445835 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:31.445837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.446618 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.446619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:31.446703 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.446707 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:31.447182 (+ 475us) client_negotiation.cc:240] Negotiation successful
0923 22:19:31.447326 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":115,"threads_started":1}
I20260923 22:19:31.447773 14710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265fcb83256e46a8a51f01ef133ab3ef" candidate_uuid: "b5732dc603724d999151a91fb9c3217c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" is_pre_election: true
I20260923 22:19:31.447949 14710 raft_consensus.cc:2454] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5732dc603724d999151a91fb9c3217c in term 0.
I20260923 22:19:31.448105 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265fcb83256e46a8a51f01ef133ab3ef" candidate_uuid: "b5732dc603724d999151a91fb9c3217c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06422fa36984b79a662e34e6c760e64" is_pre_election: true
I20260923 22:19:31.448181 14512 leader_election.cc:302] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b3b0b551f634cd6922c9ba1092a8f67, b5732dc603724d999151a91fb9c3217c; no voters:
I20260923 22:19:31.448238 14437 raft_consensus.cc:2454] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5732dc603724d999151a91fb9c3217c in term 0.
I20260923 22:19:31.448308 14795 raft_consensus.cc:2781] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:31.448351 14795 raft_consensus.cc:482] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:31.448374 14795 raft_consensus.cc:3037] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:31.449285 14795 raft_consensus.cc:504] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.449539 14795 leader_election.cc:288] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [CANDIDATE]: Term 1 election: Requested vote from peers 0b3b0b551f634cd6922c9ba1092a8f67 (127.1.129.67:35007), b06422fa36984b79a662e34e6c760e64 (127.1.129.65:36705)
I20260923 22:19:31.449602 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265fcb83256e46a8a51f01ef133ab3ef" candidate_uuid: "b5732dc603724d999151a91fb9c3217c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06422fa36984b79a662e34e6c760e64"
I20260923 22:19:31.449667 14437 raft_consensus.cc:3037] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:31.449693 14710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265fcb83256e46a8a51f01ef133ab3ef" candidate_uuid: "b5732dc603724d999151a91fb9c3217c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3b0b551f634cd6922c9ba1092a8f67"
I20260923 22:19:31.449764 14710 raft_consensus.cc:3037] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:31.450496 14437 raft_consensus.cc:2454] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5732dc603724d999151a91fb9c3217c in term 1.
I20260923 22:19:31.450730 14509 leader_election.cc:302] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06422fa36984b79a662e34e6c760e64, b5732dc603724d999151a91fb9c3217c; no voters:
I20260923 22:19:31.450834 14710 raft_consensus.cc:2454] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5732dc603724d999151a91fb9c3217c in term 1.
I20260923 22:19:31.450865 14795 raft_consensus.cc:2781] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:31.451097 14795 raft_consensus.cc:686] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 1 LEADER]: Becoming Leader. State: Replica: b5732dc603724d999151a91fb9c3217c, State: Running, Role: LEADER
I20260923 22:19:31.451222 14795 consensus_queue.cc:237] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } }
I20260923 22:19:31.452104 14290 catalog_manager.cc:5495] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c reported cstate change: term changed from 0 to 1, leader changed from <none> to b5732dc603724d999151a91fb9c3217c (127.1.129.66). New cstate: current_term: 1 leader_uuid: "b5732dc603724d999151a91fb9c3217c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5732dc603724d999151a91fb9c3217c" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20260923 22:19:31.460705 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.457347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.1.129.66:45415)
0923 22:19:31.457389 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:31.457391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:31.457401 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:19:31.457443 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:31.457444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:31.457463 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:31.457536 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:31.457739 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.458436 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.459591 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:31.459880 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:31.459948 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:31.460148 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:31.460276 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:31.460356 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:31.460608 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 22:19:31.460629 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:19:31.462567 14437 raft_consensus.cc:1261] T 265fcb83256e46a8a51f01ef133ab3ef P b06422fa36984b79a662e34e6c760e64 [term 1 FOLLOWER]: Refusing update from remote peer b5732dc603724d999151a91fb9c3217c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:31.462756 14710 raft_consensus.cc:1261] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 1 FOLLOWER]: Refusing update from remote peer b5732dc603724d999151a91fb9c3217c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:31.463150 14795 consensus_queue.cc:1035] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b06422fa36984b79a662e34e6c760e64" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:31.463451 14795 consensus_queue.cc:1035] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b3b0b551f634cd6922c9ba1092a8f67" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:31.467904 14805 mvcc.cc:205] Tried to move back new op lower bound from 7332667275107913728 to 7332667275065061376. Current Snapshot: MvccSnapshot[applied={T|T < 7332667275107913728}]
I20260923 22:19:31.468621 14290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:31.468720 14290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:31.469499 14290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aea92274bca348ef80f65d76970a3af6: Sending DeleteTablet for 3 replicas of tablet 265fcb83256e46a8a51f01ef133ab3ef
I20260923 22:19:31.469835 14539 tablet_service.cc:1514] Processing DeleteTablet for tablet 265fcb83256e46a8a51f01ef133ab3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:31 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:53180
I20260923 22:19:31.469836 14689 tablet_service.cc:1514] Processing DeleteTablet for tablet 265fcb83256e46a8a51f01ef133ab3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:31 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:58566
I20260923 22:19:31.469889 14422 tablet_service.cc:1514] Processing DeleteTablet for tablet 265fcb83256e46a8a51f01ef133ab3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:31 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:56132
I20260923 22:19:31.470060 14811 tablet_replica.cc:330] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: stopping tablet replica
I20260923 22:19:31.470150 14811 raft_consensus.cc:2229] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:19:31.470253 14811 raft_consensus.cc:2258] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:31.470381 14812 tablet_replica.cc:330] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: stopping tablet replica
I20260923 22:19:31.470472 14812 raft_consensus.cc:2229] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:31.470522 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 14328
I20260923 22:19:31.470541 14812 raft_consensus.cc:2258] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:31.472513 14811 ts_tablet_manager.cc:1905] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:31.474328 14811 ts_tablet_manager.cc:1918] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:31.474406 14811 log.cc:1196] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/265fcb83256e46a8a51f01ef133ab3ef
I20260923 22:19:31.474713 14811 ts_tablet_manager.cc:1939] T 265fcb83256e46a8a51f01ef133ab3ef P b5732dc603724d999151a91fb9c3217c: Deleting consensus metadata
I20260923 22:19:31.475123 14276 catalog_manager.cc:4841] TS b5732dc603724d999151a91fb9c3217c (127.1.129.66:45415): tablet 265fcb83256e46a8a51f01ef133ab3ef (table test-table [id=c9c9e650d6134e0aa98cc84255e6d7f7]) successfully deleted
W20260923 22:19:31.475967 14274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:36705 (error 108)
W20260923 22:19:31.475970 14509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:36705 (error 108)
W20260923 22:19:31.476467 14798 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:19:31.476350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:36705 (local address 127.1.129.66:53143)
0923 22:19:31.476395 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:31.476431 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:36705: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 22:19:31.477063 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 14492
I20260923 22:19:31.477700 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:31.477111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:36705 (local address 127.0.0.1:56138)
0923 22:19:31.477622 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:31.477662 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:36705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
W20260923 22:19:31.479723 14274 catalog_manager.cc:4584] TS b06422fa36984b79a662e34e6c760e64 (127.1.129.65:36705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 265fcb83256e46a8a51f01ef133ab3ef: Network error: Client connection negotiation failed: client connection to 127.1.129.65:36705: connect: Connection refused (error 111)
I20260923 22:19:31.480428 14812 ts_tablet_manager.cc:1905] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:31.482055 14812 ts_tablet_manager.cc:1918] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:31.482165 14812 log.cc:1196] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/265fcb83256e46a8a51f01ef133ab3ef
I20260923 22:19:31.482450 14812 ts_tablet_manager.cc:1939] T 265fcb83256e46a8a51f01ef133ab3ef P 0b3b0b551f634cd6922c9ba1092a8f67: Deleting consensus metadata
I20260923 22:19:31.482976 14277 catalog_manager.cc:4841] TS 0b3b0b551f634cd6922c9ba1092a8f67 (127.1.129.67:35007): tablet 265fcb83256e46a8a51f01ef133ab3ef (table test-table [id=c9c9e650d6134e0aa98cc84255e6d7f7]) successfully deleted
I20260923 22:19:31.487352 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 14625
I20260923 22:19:31.496488 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 14257
I20260923 22:19:31.502564 1541 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 22:19:31.837288 1541 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 22:19:31.837370 1541 mini_ranger.cc:66] Stopping Ranger...
I20260923 22:19:32.182142 1541 mini_ranger.cc:68] Stopped Ranger
2026-09-23 22:19:32.182 UTC [13324] LOG: received smart shutdown request
2026-09-23 22:19:32.186 UTC [13324] LOG: background worker "logical replication launcher" (PID 13333) exited with exit code 1
2026-09-23 22:19:32.186 UTC [13328] LOG: shutting down
2026-09-23 22:19:32.357 UTC [13324] LOG: database system is shut down
2026-09-23T22:19:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48047 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:19:32 dist-test-slave-dmw7 krb5kdc[14824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:19:32 dist-test-slave-dmw7 krb5kdc[14824](info): set up 2 sockets
Sep 23 22:19:32 dist-test-slave-dmw7 krb5kdc[14824](info): commencing operation
krb5kdc: starting...
W20260923 22:19:33.510564 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:19:33 dist-test-slave-dmw7 krb5kdc[14824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:19:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:19:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:19:33.627363 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36567
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36567,127.1.129.125:45087,127.1.129.124:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:33.707024 14837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:33.707221 14837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:33.707242 14837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:33.708352 14837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:33.708391 14837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:33.708406 14837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:33.708419 14837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:33.708465 14837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:33.709921 14837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36567,127.1.129.125:45087,127.1.129.124:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:33.710150 14837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:33.710410 14837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:33.715071 14842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.715183 14843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:33.715220 14837 server_base.cc:1034] running on GCE node
W20260923 22:19:33.715071 14845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:33.715617 14837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:33.715938 14837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:33.717104 14837 hybrid_clock.cc:648] HybridClock initialized: now 1790201973717091 us; error 42 us; skew 500 ppm
Sep 23 22:19:33 dist-test-slave-dmw7 krb5kdc[14824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:33.719532 14837 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:19:33.719898 14837 webserver.cc:460] Webserver started at http://127.1.129.126:41277/ using document root <none> and password file <none>
I20260923 22:19:33.720081 14837 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:33.720121 14837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:33.720207 14837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:33.721175 14837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "ce1c2ecd1b404bf0868e49f0577615b4"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-dmw7"
server_key: "dae731fed7074013d45b5fa790600765"
server_key_iv: "be9b5647cffb9ffe63d2f9580288d771"
server_key_version: "encryptionkey@0"
I20260923 22:19:33.721531 14837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "ce1c2ecd1b404bf0868e49f0577615b4"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-dmw7"
server_key: "dae731fed7074013d45b5fa790600765"
server_key_iv: "be9b5647cffb9ffe63d2f9580288d771"
server_key_version: "encryptionkey@0"
I20260923 22:19:33.723152 14837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 22:19:33.723876 14852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.724048 14837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:33.724136 14837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "ce1c2ecd1b404bf0868e49f0577615b4"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-dmw7"
server_key: "dae731fed7074013d45b5fa790600765"
server_key_iv: "be9b5647cffb9ffe63d2f9580288d771"
server_key_version: "encryptionkey@0"
I20260923 22:19:33.724188 14837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:33.743239 14837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:33.745289 14837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:33.745409 14837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:33.749295 14837 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:36567
I20260923 22:19:33.749363 14904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:36567 every 8 connection(s)
I20260923 22:19:33.749701 14837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:33.750590 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:33.751991 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 14837
I20260923 22:19:33.752115 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:19:33.752297 1541 external_mini_cluster.cc:1428] Setting key f0cd1bd4fd2d6a39fe71758dba4a2d4f
I20260923 22:19:33.752827 14905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.129.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.125@KRBTEST.COM" created.
Sep 23 22:19:33 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:33.765894 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.753668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36567 (local address 127.0.0.1:36246)
0923 22:19:33.754735 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.754779 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:33.755269 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:33.755566 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:33.755569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:33.755852 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:33.756001 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.756005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.757050 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.757051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.757720 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.757721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.758221 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.758222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.758287 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.758839 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:33.758848 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:33.762447 (+ 3599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:33.764483 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.764490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.764492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.764677 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.764907 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.764908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.764908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.764992 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.765333 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:33.765337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:33.765396 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 22:19:33.765494 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 22:19:33.765713 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":83,"thread_start_us":90,"threads_started":1}
I20260923 22:19:33.765897 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.753702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.1.129.126:36567)
0923 22:19:33.754734 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:33.754744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:33.754879 (+ 135us) server_negotiation.cc:407] Connection header received
0923 22:19:33.755422 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:33.755425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:33.755444 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:33.755559 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:33.756111 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.756932 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.757810 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.758133 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.758240 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.762552 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:33.762559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:33.762560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:33.762567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:33.764394 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.764765 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.764768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.764768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.764844 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.765053 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.765054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.765054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.765261 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:33.765343 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:33.765519 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 22:19:33.765654 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":66,"threads_started":1}
I20260923 22:19:33.766569 14905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:33.766968 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.766861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56350)
0923 22:19:33.766908 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.766948 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 22:19:33.767117 14854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
I20260923 22:19:33.767490 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.767407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56356)
0923 22:19:33.767446 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.767470 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260923 22:19:33.767586 14905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:45087: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab.
Entry for principal kudu/127.1.129.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab.
I20260923 22:19:33.775478 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.125:45087
--webserver_interface=127.1.129.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab
--principal=kudu/127.1.129.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36567,127.1.129.125:45087,127.1.129.124:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:33.816839 14905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } attempt: 1
I20260923 22:19:33.817479 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.817307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56364)
0923 22:19:33.817377 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.817424 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 22:19:33.817883 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.817794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56366)
0923 22:19:33.817835 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.817864 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 22:19:33.818007 14905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:45087: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
W20260923 22:19:33.861081 14912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:33.861323 14912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:33.861349 14912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:33.862499 14912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:33.862540 14912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:33.862555 14912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:33.862567 14912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:33.862608 14912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:33.864152 14912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36567,127.1.129.125:45087,127.1.129.124:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab
--principal=kudu/127.1.129.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.125:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.129.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:33.864398 14912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:33.864621 14912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:33.869443 14918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.869575 14917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.869450 14920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:33.869498 14912 server_base.cc:1034] running on GCE node
I20260923 22:19:33.869961 14912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:33.870258 14912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:33.871412 14912 hybrid_clock.cc:648] HybridClock initialized: now 1790201973871403 us; error 38 us; skew 500 ppm
Sep 23 22:19:33 dist-test-slave-dmw7 krb5kdc[14824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:33.873909 14912 init.cc:377] Logged in from keytab as kudu/127.1.129.125@KRBTEST.COM (short username kudu)
I20260923 22:19:33.874408 14912 webserver.cc:460] Webserver started at http://127.1.129.125:45159/ using document root <none> and password file <none>
I20260923 22:19:33.874725 14912 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:33.874794 14912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:33.874934 14912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:33.876029 14912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/data/instance:
uuid: "da1bd184e3e44cfc8e7cd55bcb95295f"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-dmw7"
server_key: "cb91069dab0f623230f6dbcd542e1403"
server_key_iv: "1d79fa77dcd8aec90a8e21eba2a3a79f"
server_key_version: "encryptionkey@0"
I20260923 22:19:33.876412 14912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/wal/instance:
uuid: "da1bd184e3e44cfc8e7cd55bcb95295f"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-dmw7"
server_key: "cb91069dab0f623230f6dbcd542e1403"
server_key_iv: "1d79fa77dcd8aec90a8e21eba2a3a79f"
server_key_version: "encryptionkey@0"
I20260923 22:19:33.877801 14912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:19:33.878602 14927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.878836 14912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:33.878932 14912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/wal
uuid: "da1bd184e3e44cfc8e7cd55bcb95295f"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-dmw7"
server_key: "cb91069dab0f623230f6dbcd542e1403"
server_key_iv: "1d79fa77dcd8aec90a8e21eba2a3a79f"
server_key_version: "encryptionkey@0"
I20260923 22:19:33.879015 14912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:33.886152 14905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } attempt: 2
I20260923 22:19:33.886565 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.886426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56378)
0923 22:19:33.886485 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.886531 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 22:19:33.886943 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.886842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56382)
0923 22:19:33.886892 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.886921 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 22:19:33.887065 14905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:45087: Network error: Client connection negotiation failed: client connection to 127.1.129.125:45087: connect: Connection refused (error 111)
I20260923 22:19:33.888259 14912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:33.889968 14912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:33.890125 14912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:33.894241 14912 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.125:45087
I20260923 22:19:33.894274 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.125:45087 every 8 connection(s)
I20260923 22:19:33.894667 14912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
I20260923 22:19:33.895478 14980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:33.896981 14980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:33.900024 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 14912
I20260923 22:19:33.900144 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-1/wal/instance
I20260923 22:19:33.900316 1541 external_mini_cluster.cc:1428] Setting key e1bb2cb7812548181adcf1e77e043e29
Sep 23 22:19:33 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:33.906387 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.897664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.1.129.126:36567)
0923 22:19:33.897728 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:33.897731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:33.897963 (+ 232us) server_negotiation.cc:407] Connection header received
0923 22:19:33.898424 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:33.898425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:33.898437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:33.898494 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:33.899180 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.899680 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.900709 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.901051 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.901129 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.903545 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:33.903547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:33.903547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:33.903555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:33.904986 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.905409 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.905411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.905411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.905474 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.905735 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.905736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.905736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.905853 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:33.905924 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:33.906203 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 22:19:33.906291 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:19:33.906605 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.897651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36567 (local address 127.0.0.1:36248)
0923 22:19:33.897844 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.897882 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:33.898341 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:33.898568 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:33.898572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:33.898890 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:33.899085 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.899099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.899760 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.899761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.900628 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.900630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.901136 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.901137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.901233 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.901805 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:33.901816 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:33.903445 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:33.905085 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.905095 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.905098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.905335 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.905539 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.905540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.905541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.905660 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.905916 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:33.905922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:33.906073 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 22:19:33.906183 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 22:19:33.906352 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":105,"threads_started":1}
I20260923 22:19:33.907296 14980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.129.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.124@KRBTEST.COM" created.
Sep 23 22:19:33 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:19:33.914937 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.907637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.1.129.125:45087)
0923 22:19:33.907793 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:33.907797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:33.907813 (+ 16us) server_negotiation.cc:407] Connection header received
0923 22:19:33.907829 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:33.907831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:33.907841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:33.907917 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:33.908190 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.908923 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.909654 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.909983 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.910067 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.912056 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:33.912065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:33.912066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:33.912081 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:33.913671 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.914101 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.914105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.914105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.914157 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.914358 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.914359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.914359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.914497 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:33.914559 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:33.914785 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 22:19:33.914859 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260923 22:19:33.915086 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.907580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56388)
0923 22:19:33.907631 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.907641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:33.907680 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:33.907917 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:33.907918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:33.908064 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:33.908109 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.908111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.909021 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.909022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.909566 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.909568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.910065 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.910066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.910136 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.910585 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:33.910591 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:33.911971 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:33.913760 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.913761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.913762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.914012 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.914224 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.914225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.914226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.914296 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.914564 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:33.914565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:33.914652 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 22:19:33.914751 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 22:19:33.914942 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":72}
I20260923 22:19:33.916007 14980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:33.916364 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.916241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43178)
0923 22:19:33.916288 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.916339 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:19:33.916504 14931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
I20260923 22:19:33.916958 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.916854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43182)
0923 22:19:33.916900 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.916930 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:19:33.917064 14980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:45369: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab.
Entry for principal kudu/127.1.129.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab.
I20260923 22:19:33.925927 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.124:45369
--webserver_interface=127.1.129.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab
--principal=kudu/127.1.129.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36567,127.1.129.125:45087,127.1.129.124:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:33.966321 14980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } attempt: 1
I20260923 22:19:33.966917 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.966772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43184)
0923 22:19:33.966834 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.966882 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 22:19:33.967334 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.967250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43198)
0923 22:19:33.967289 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.967315 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 22:19:33.967458 14980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:45369: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
I20260923 22:19:33.978245 14905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } attempt: 3
Sep 23 22:19:33 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:19:33.985499 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.978702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.1.129.125:45087)
0923 22:19:33.978770 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:33.978772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:33.978785 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:19:33.978815 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:33.978816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:33.978827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:33.978897 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:33.979241 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.979731 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.980412 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.980640 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.980714 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.982890 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:33.982892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:33.982893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:33.982901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:33.984330 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.984717 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.984718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.984718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.984780 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.984978 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.984978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.984979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.985093 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:33.985166 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:33.985325 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 22:19:33.985398 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:19:33.985496 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.978622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56398)
0923 22:19:33.978687 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.978703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:33.978758 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:33.978914 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:33.978915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:33.979085 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:33.979151 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.979152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.979828 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.979830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.980332 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.980334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.980711 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.980712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.980790 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.981290 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:33.981294 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:33.982807 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:33.984455 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.984456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.984457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.984640 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.984846 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.984847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.984847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.984906 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.985156 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:33.985157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:33.985208 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 22:19:33.985285 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 22:19:33.985391 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 22:19:33.985872 14905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:33.986238 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.986135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43206)
0923 22:19:33.986183 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.986210 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 22:19:33.986685 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.986516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43218)
0923 22:19:33.986578 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.986610 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 22:19:33.986778 14905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:45369: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
W20260923 22:19:34.009433 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:34.009707 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:34.009748 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:34.011089 14987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:34.011147 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:34.011168 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:34.011189 14987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:34.011231 14987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:34.012781 14987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36567,127.1.129.125:45087,127.1.129.124:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab
--principal=kudu/127.1.129.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.124:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.129.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:34.013064 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:34.013288 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:19:34.017982 14905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } attempt: 1
W20260923 22:19:34.018308 14993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:34.018492 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.018348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43226)
0923 22:19:34.018430 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.018457 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 22:19:34.018421 14987 server_base.cc:1034] running on GCE node
W20260923 22:19:34.018563 14995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:34.018596 14992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:34.018872 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.018756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43230)
0923 22:19:34.018809 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.018840 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 22:19:34.018924 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260923 22:19:34.018998 14905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:45369: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
I20260923 22:19:34.019228 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:34.020396 14987 hybrid_clock.cc:648] HybridClock initialized: now 1790201974020379 us; error 27 us; skew 500 ppm
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:34.023065 14987 init.cc:377] Logged in from keytab as kudu/127.1.129.124@KRBTEST.COM (short username kudu)
I20260923 22:19:34.023532 14987 webserver.cc:460] Webserver started at http://127.1.129.124:39631/ using document root <none> and password file <none>
I20260923 22:19:34.023764 14987 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:34.023828 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:34.023975 14987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:34.025054 14987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/data/instance:
uuid: "7ac6bffae9174caebd232b87dfe92685"
format_stamp: "Formatted at 2026-09-23 22:19:34 on dist-test-slave-dmw7"
server_key: "9388a54fd512987d9b6d727d10f6e27f"
server_key_iv: "89dfe84145cae9f1989d467b0460057c"
server_key_version: "encryptionkey@0"
I20260923 22:19:34.025417 14987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/wal/instance:
uuid: "7ac6bffae9174caebd232b87dfe92685"
format_stamp: "Formatted at 2026-09-23 22:19:34 on dist-test-slave-dmw7"
server_key: "9388a54fd512987d9b6d727d10f6e27f"
server_key_iv: "89dfe84145cae9f1989d467b0460057c"
server_key_version: "encryptionkey@0"
I20260923 22:19:34.026906 14987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:19:34.027729 15002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.027906 14987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:34.028009 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/wal
uuid: "7ac6bffae9174caebd232b87dfe92685"
format_stamp: "Formatted at 2026-09-23 22:19:34 on dist-test-slave-dmw7"
server_key: "9388a54fd512987d9b6d727d10f6e27f"
server_key_iv: "89dfe84145cae9f1989d467b0460057c"
server_key_version: "encryptionkey@0"
I20260923 22:19:34.028084 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:34.035631 14980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } attempt: 2
I20260923 22:19:34.036080 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.035948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43232)
0923 22:19:34.036012 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.036050 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 22:19:34.036437 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.036339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43248)
0923 22:19:34.036386 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.036411 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 22:19:34.036536 14980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:45369: Network error: Client connection negotiation failed: client connection to 127.1.129.124:45369: connect: Connection refused (error 111)
I20260923 22:19:34.046494 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:34.048942 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:34.049062 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:34.053177 14987 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.124:45369
I20260923 22:19:34.053231 15054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.124:45369 every 8 connection(s)
I20260923 22:19:34.053583 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
I20260923 22:19:34.054431 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:34.056068 15055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:34.061147 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 14987
I20260923 22:19:34.061260 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201819212015-1541-0/minicluster-data/master-2/wal/instance
I20260923 22:19:34.061510 1541 external_mini_cluster.cc:1428] Setting key b9a28f65ff38b257b14758573adcc855
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:34.065722 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.056816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.1.129.126:36567)
0923 22:19:34.056886 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.056888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.057160 (+ 272us) server_negotiation.cc:407] Connection header received
0923 22:19:34.057675 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.057676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.057687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.057755 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.058340 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.058905 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.059854 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.060065 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.060127 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.062574 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.062575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.062575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.062582 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.064045 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.064586 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.064588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.064588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.064667 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.064949 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.064950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.064950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.065096 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.065166 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.065549 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.065644 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:19:34.065902 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.056790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36567 (local address 127.0.0.1:36260)
0923 22:19:34.057025 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.057063 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:34.057546 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:34.057808 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:34.057812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:34.058060 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:34.058250 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.058254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.058995 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.058996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.059770 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.059772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.060138 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.060139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.060238 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.060825 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:34.060833 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:34.062491 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:34.064119 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.064126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.064128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.064442 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.064762 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.064763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.064763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.064876 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.065171 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:34.065178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:34.065352 (+ 174us) client_negotiation.cc:769] Sending connection context
0923 22:19:34.065468 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 22:19:34.065650 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":112,"threads_started":1}
I20260923 22:19:34.066728 15055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:19:34.070884 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.061891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.1.129.124:45369)
0923 22:19:34.062088 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.062093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.062138 (+ 45us) server_negotiation.cc:407] Connection header received
0923 22:19:34.062159 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.062161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.062181 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.062243 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.062983 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.063656 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.064586 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.065046 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.065121 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.067690 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.067697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.067698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.067704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.069452 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.069891 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.069895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.069895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.069977 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.070179 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.070180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.070180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.070370 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.070526 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.070749 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.070812 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:19:34.074852 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.066974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:45087 (local address 127.0.0.1:56404)
0923 22:19:34.067019 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.067030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:34.067065 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:34.067224 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:34.067225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:34.067385 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:34.067431 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.067432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.068165 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.068166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.068951 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.068954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.069299 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.069300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.069382 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.069878 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:34.069883 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:34.071477 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:34.073825 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.073826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.073826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.073981 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.074191 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.074192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.074192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.074256 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.074494 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:34.074494 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:34.074541 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 22:19:34.074616 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 22:19:34.074752 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 22:19:34.074880 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.067059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.1.129.125:45087)
0923 22:19:34.067113 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.067116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.067123 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:34.067147 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.067148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.067157 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.067222 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.067540 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.068087 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.069045 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.069225 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.069283 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.071573 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.071575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.071575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.071583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.073742 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.074044 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.074045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.074046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.074108 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.074316 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.074317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.074317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.074429 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.074502 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.074741 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.074819 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:19:34.075250 15055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:19:34.079771 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.071636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.1.129.126:36567)
0923 22:19:34.071682 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.071684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.071734 (+ 50us) server_negotiation.cc:407] Connection header received
0923 22:19:34.071811 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.071812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.071820 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.071861 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.072492 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.073145 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.074085 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.074259 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.074348 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.076974 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.076976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.076976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.076982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.078488 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.078890 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.078891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.078892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.078946 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.079150 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.079151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.079152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.079263 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.079384 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.079583 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.079663 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:19:34.082012 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.075529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43264)
0923 22:19:34.075559 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.075567 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:34.075599 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:34.075729 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:34.075730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:34.075843 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:34.075871 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.075872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.076568 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.076569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.077261 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.077263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.077627 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.077628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.077676 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.078204 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:34.078208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:34.079322 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:34.080984 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.080985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.080986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.081146 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.081343 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.081344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.081344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.081396 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.081658 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:34.081659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:34.081697 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 22:19:34.081768 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 22:19:34.081882 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260923 22:19:34.082055 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.075567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.1.129.124:45369)
0923 22:19:34.075603 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.075605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.075635 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:19:34.075656 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.075656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.075668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.075729 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.075938 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.076447 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.077345 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.077553 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.077631 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.079420 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.079421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.079421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.079427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.080899 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.081211 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.081212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.081212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.081283 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.081460 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.081461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.081461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.081603 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.081665 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.081800 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.081930 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":20}
I20260923 22:19:34.083498 15055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: Bootstrap starting.
I20260923 22:19:34.084372 15055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:34.084673 15055 log.cc:824] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: Log is configured to *not* fsync() on all Append() calls
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:19:34.085523 15055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: No bootstrap required, opened a new log
I20260923 22:19:34.087563 15055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.087677 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.080750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.1.129.125:45087)
0923 22:19:34.080811 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.080813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.080820 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:34.080841 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.080842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.080851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.080911 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.081415 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.081887 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.082772 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.082974 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.083058 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.085303 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.085304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.085304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.085310 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.086619 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.087019 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.087020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.087021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.087071 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.087259 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.087259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.087260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.087352 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.087487 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.087557 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.087616 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:19:34.087723 15055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:34.087765 15055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac6bffae9174caebd232b87dfe92685, State: Initialized, Role: FOLLOWER
I20260923 22:19:34.087915 15055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.088243 15055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:34.088253 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.088379 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:34.089113 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 22:19:34.090139 15072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:34.090202 15072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:34.090443 14987 master_runner.cc:186] Error getting master registration for 127.1.129.126:36567: OK, instance_id { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" instance_seqno: 1790201973748198 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:19:34.092687 14987 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:19:34.094444 14905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } attempt: 2
I20260923 22:19:34.099279 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.090781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.1.129.126:36567)
0923 22:19:34.090818 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.090821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.090930 (+ 109us) server_negotiation.cc:407] Connection header received
0923 22:19:34.091026 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.091027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.091036 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.091064 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.091797 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.092489 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.093496 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.093768 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.093828 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.095602 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.095603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.095604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.095610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.097611 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.098031 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.098033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.098033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.098098 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.098314 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.098315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.098316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.098453 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.098565 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.099083 (+ 518us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.099186 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:19:34.101069 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.090695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.1.129.124:45369)
0923 22:19:34.090742 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.090744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.093740 (+ 2996us) server_negotiation.cc:407] Connection header received
0923 22:19:34.093831 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.093832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.093841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.093870 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.094597 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.095310 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.096302 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.096595 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.096638 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.097994 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.097995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.097996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.098003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.099693 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.100123 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.100124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.100125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.100179 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.100384 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.100384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.100385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.100575 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.100667 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.100901 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.100983 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:19:34.101079 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.090692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.1.129.125:45087)
0923 22:19:34.090728 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.090730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.093606 (+ 2876us) server_negotiation.cc:407] Connection header received
0923 22:19:34.093653 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.093654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.093664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.093721 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.094369 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.095048 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.095994 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.096194 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.096265 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.097613 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.097614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.097615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.097621 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.099665 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.100112 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.100113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.100113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.100183 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.100430 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.100431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.100431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.100549 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.100661 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.100901 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.100981 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:19:34.107857 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.094715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43280)
0923 22:19:34.094802 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.094811 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:34.094845 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:34.101371 (+ 6526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:34.101372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:34.101477 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:34.101508 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.101510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.102399 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.102400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.103234 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.103237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.103551 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.103552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.103608 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.103953 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:34.103957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:34.105218 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:34.106809 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.106810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.106810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.106987 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.107206 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.107206 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.107207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.107268 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.107547 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:34.107548 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:34.107589 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 22:19:34.107666 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 22:19:34.107760 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 22:19:34.107857 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.094904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.1.129.124:45369)
0923 22:19:34.101142 (+ 6238us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.101144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.101152 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:19:34.101171 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.101172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.101181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.101236 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.101595 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.102304 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.103310 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.103482 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.103534 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.105306 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.105307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.105307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.105314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.106715 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.107078 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.107079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.107080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.107138 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.107335 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.107336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.107336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.107461 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.107519 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.107704 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.107771 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6224,"thread_start_us":83,"threads_started":1}
I20260923 22:19:34.109125 14905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4: Bootstrap starting.
I20260923 22:19:34.109751 14905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:34.110025 14905 log.cc:824] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:34.110908 14905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4: No bootstrap required, opened a new log
I20260923 22:19:34.112428 14905 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.112542 14905 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:34.112566 14905 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1c2ecd1b404bf0868e49f0577615b4, State: Initialized, Role: FOLLOWER
I20260923 22:19:34.112703 14905 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.112988 15083 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.113000 14905 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:34.114436 15083 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:19:34.114928 15094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:34.114981 15094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:34.115628 14837 master_runner.cc:186] Error getting master registration for 127.1.129.125:45087: OK, instance_id { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" instance_seqno: 1790201973893146 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:19:34.118407 14837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:19:34.130666 14980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } attempt: 3
Sep 23 22:19:34 dist-test-slave-dmw7 krb5kdc[14824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:19:34.139962 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.131005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:45369 (local address 127.0.0.1:43286)
0923 22:19:34.131085 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.131095 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:34.131132 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:34.131430 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:34.131430 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:34.131614 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:34.131666 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.131669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.132554 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.132555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.133325 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.133327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.133834 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.133835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.133889 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.135276 (+ 1387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:34.135281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:34.136932 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:34.138797 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.138798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.138799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.139019 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.139235 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.139236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.139236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.139303 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.139625 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:34.139627 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:34.139677 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 22:19:34.139743 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:19:34.139850 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 22:19:34.140349 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.131227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.1.129.124:45369)
0923 22:19:34.131298 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.131300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.131307 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:34.131329 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.131330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.131339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.131383 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.131742 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.132462 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.133418 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.133721 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.133780 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.137005 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.137006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.137006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.137015 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.138706 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.139099 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.139100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.139101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.139162 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.139369 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.139370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.139370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.139508 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.139592 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.140184 (+ 592us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.140258 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 22:19:34.142266 14980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: Bootstrap starting.
I20260923 22:19:34.143091 14980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:34.143405 14980 log.cc:824] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:34.144311 14980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: No bootstrap required, opened a new log
I20260923 22:19:34.147668 14980 raft_consensus.cc:348] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.147831 14980 raft_consensus.cc:374] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:34.147877 14980 raft_consensus.cc:729] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1bd184e3e44cfc8e7cd55bcb95295f, State: Initialized, Role: FOLLOWER
I20260923 22:19:34.148008 14980 consensus_queue.cc:260] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.148350 14980 sys_catalog.cc:564] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:34.148310 15097 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.148499 15097 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:19:34.150281 15108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:34.150393 15108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:34.152503 14912 master_runner.cc:263] No leader master found from master da1bd184e3e44cfc8e7cd55bcb95295f
I20260923 22:19:34.152558 14912 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 22:19:34.246260 14858 tablet.cc:2369] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:19:34.299690 15008 tablet.cc:2369] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:34.386729 15083 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:34.386888 15083 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.387187 15083 leader_election.cc:288] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da1bd184e3e44cfc8e7cd55bcb95295f (127.1.129.125:45087), 7ac6bffae9174caebd232b87dfe92685 (127.1.129.124:45369)
I20260923 22:19:34.387501 15029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac6bffae9174caebd232b87dfe92685" is_pre_election: true
I20260923 22:19:34.387532 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" is_pre_election: true
I20260923 22:19:34.387674 14954 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1c2ecd1b404bf0868e49f0577615b4 in term 0.
I20260923 22:19:34.387676 15029 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1c2ecd1b404bf0868e49f0577615b4 in term 0.
I20260923 22:19:34.387883 14856 leader_election.cc:302] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac6bffae9174caebd232b87dfe92685, ce1c2ecd1b404bf0868e49f0577615b4; no voters:
I20260923 22:19:34.388082 15083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:34.388149 15083 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:34.388182 15083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:34.389034 15083 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.389168 15083 leader_election.cc:288] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [CANDIDATE]: Term 1 election: Requested vote from peers da1bd184e3e44cfc8e7cd55bcb95295f (127.1.129.125:45087), 7ac6bffae9174caebd232b87dfe92685 (127.1.129.124:45369)
I20260923 22:19:34.389343 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f"
I20260923 22:19:34.389343 15029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac6bffae9174caebd232b87dfe92685"
I20260923 22:19:34.389451 14954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:34.389451 15029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:34.390230 15029 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1c2ecd1b404bf0868e49f0577615b4 in term 1.
I20260923 22:19:34.390272 14954 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1c2ecd1b404bf0868e49f0577615b4 in term 1.
I20260923 22:19:34.390427 14854 leader_election.cc:302] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce1c2ecd1b404bf0868e49f0577615b4, da1bd184e3e44cfc8e7cd55bcb95295f; no voters:
I20260923 22:19:34.390537 15083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:34.390697 15083 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [term 1 LEADER]: Becoming Leader. State: Replica: ce1c2ecd1b404bf0868e49f0577615b4, State: Running, Role: LEADER
W20260923 22:19:34.390838 14933 tablet.cc:2369] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:34.390825 15083 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } }
I20260923 22:19:34.391198 15110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce1c2ecd1b404bf0868e49f0577615b4. Latest consensus state: current_term: 1 leader_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.391283 15110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:34.391487 15112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:34.391667 15112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:34.392966 14954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [term 1 FOLLOWER]: Refusing update from remote peer ce1c2ecd1b404bf0868e49f0577615b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:34.393003 15029 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [term 1 FOLLOWER]: Refusing update from remote peer ce1c2ecd1b404bf0868e49f0577615b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:34.393285 15083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:34.393421 15110 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:34.394583 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce1c2ecd1b404bf0868e49f0577615b4. Latest consensus state: current_term: 1 leader_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.394693 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:34.394677 15097 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce1c2ecd1b404bf0868e49f0577615b4. Latest consensus state: current_term: 1 leader_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.394760 15097 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:34.395537 15083 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.395607 15083 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:34.395658 15118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.395735 15118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:34.395706 15097 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.395792 15097 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:34.395984 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1c2ecd1b404bf0868e49f0577615b4" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36567 } } peers { permanent_uuid: "da1bd184e3e44cfc8e7cd55bcb95295f" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 45087 } } peers { permanent_uuid: "7ac6bffae9174caebd232b87dfe92685" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 45369 } } }
I20260923 22:19:34.396064 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:34.396144 15112 catalog_manager.cc:1347] Generated new cluster ID: 9921ff37584b4baa9451bf0024f2e7c2
I20260923 22:19:34.396189 15112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:19:34.396185 15115 mvcc.cc:205] Tried to move back new op lower bound from 7332667287111430144 to 7332667287105388544. Current Snapshot: MvccSnapshot[applied={T|T < 7332667287111430144}]
I20260923 22:19:34.396257 15113 mvcc.cc:205] Tried to move back new op lower bound from 7332667287111430144 to 7332667287105388544. Current Snapshot: MvccSnapshot[applied={T|T < 7332667287111430144 or (T in {7332667287111430144})}]
I20260923 22:19:34.401906 15112 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:34.402565 15112 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:34.405776 15112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce1c2ecd1b404bf0868e49f0577615b4: Generated new TSK 0
I20260923 22:19:34.405962 15112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:19:35.090708 15072 catalog_manager.cc:1259] Loaded cluster ID: 9921ff37584b4baa9451bf0024f2e7c2
I20260923 22:19:35.090802 15072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: loading cluster ID for follower catalog manager: success
I20260923 22:19:35.091615 15072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: acquiring CA information for follower catalog manager: success
I20260923 22:19:35.091904 15072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ac6bffae9174caebd232b87dfe92685: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:19:35.150943 15108 catalog_manager.cc:1259] Loaded cluster ID: 9921ff37584b4baa9451bf0024f2e7c2
I20260923 22:19:35.151024 15108 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: loading cluster ID for follower catalog manager: success
I20260923 22:19:35.151875 15108 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: acquiring CA information for follower catalog manager: success
I20260923 22:19:35.152202 15108 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da1bd184e3e44cfc8e7cd55bcb95295f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:19:35.469664 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 14837
I20260923 22:19:35.475225 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 14912
I20260923 22:19:35.480139 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 14987
2026-09-23T22:19:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3009 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:19:35 dist-test-slave-dmw7 krb5kdc[15127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:19:35 dist-test-slave-dmw7 krb5kdc[15127](info): set up 2 sockets
Sep 23 22:19:35 dist-test-slave-dmw7 krb5kdc[15127](info): commencing operation
krb5kdc: starting...
W20260923 22:19:36.510741 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:19:36 dist-test-slave-dmw7 krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:19:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:19:36.629879 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44401
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:44401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:36.710600 15140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:36.710893 15140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:36.710933 15140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:36.712162 15140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:36.712239 15140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:36.712262 15140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:36.712284 15140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:36.712312 15140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:36.713819 15140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:44401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:36.714097 15140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:36.714342 15140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:36.719228 15145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:36.719260 15148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:36.719262 15146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:36.719558 15140 server_base.cc:1034] running on GCE node
I20260923 22:19:36.719801 15140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:36.720108 15140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:36.721271 15140 hybrid_clock.cc:648] HybridClock initialized: now 1790201976721250 us; error 29 us; skew 500 ppm
Sep 23 22:19:36 dist-test-slave-dmw7 krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:36.723842 15140 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:19:36.724299 15140 webserver.cc:460] Webserver started at http://127.1.129.126:38863/ using document root <none> and password file <none>
I20260923 22:19:36.724541 15140 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:36.724586 15140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:36.724680 15140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:36.725705 15140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "6678b6b56a074a27951450ff508f8fee"
format_stamp: "Formatted at 2026-09-23 22:19:36 on dist-test-slave-dmw7"
server_key: "8bdbc5c34eea4dbc776c50d813add125"
server_key_iv: "58d4182b3d2081c15867406eaa13d408"
server_key_version: "encryptionkey@0"
I20260923 22:19:36.726011 15140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "6678b6b56a074a27951450ff508f8fee"
format_stamp: "Formatted at 2026-09-23 22:19:36 on dist-test-slave-dmw7"
server_key: "8bdbc5c34eea4dbc776c50d813add125"
server_key_iv: "58d4182b3d2081c15867406eaa13d408"
server_key_version: "encryptionkey@0"
I20260923 22:19:36.727415 15140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:19:36.728186 15155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:36.728387 15140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:36.728493 15140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "6678b6b56a074a27951450ff508f8fee"
format_stamp: "Formatted at 2026-09-23 22:19:36 on dist-test-slave-dmw7"
server_key: "8bdbc5c34eea4dbc776c50d813add125"
server_key_iv: "58d4182b3d2081c15867406eaa13d408"
server_key_version: "encryptionkey@0"
I20260923 22:19:36.728572 15140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:36.759245 15140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:36.761615 15140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:36.761781 15140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:36.765673 15140 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:44401
I20260923 22:19:36.765715 15207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:44401 every 8 connection(s)
I20260923 22:19:36.766076 15140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:36.767017 15208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:36.769302 15208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee: Bootstrap starting.
I20260923 22:19:36.769817 15208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:36.770047 15208 log.cc:824] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:36.770768 15208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee: No bootstrap required, opened a new log
I20260923 22:19:36.772262 15208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6678b6b56a074a27951450ff508f8fee" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44401 } }
I20260923 22:19:36.772398 15208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:36.772420 15208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6678b6b56a074a27951450ff508f8fee, State: Initialized, Role: FOLLOWER
I20260923 22:19:36.772553 15208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6678b6b56a074a27951450ff508f8fee" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44401 } }
I20260923 22:19:36.772631 15208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:36.772655 15208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:36.772683 15208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:36.773159 15208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6678b6b56a074a27951450ff508f8fee" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44401 } }
I20260923 22:19:36.773241 15208 leader_election.cc:302] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6678b6b56a074a27951450ff508f8fee; no voters:
I20260923 22:19:36.773380 15208 leader_election.cc:288] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:19:36.773406 15211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:36.773523 15211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [term 1 LEADER]: Becoming Leader. State: Replica: 6678b6b56a074a27951450ff508f8fee, State: Running, Role: LEADER
I20260923 22:19:36.773643 15208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:36.773681 15211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6678b6b56a074a27951450ff508f8fee" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44401 } }
I20260923 22:19:36.774263 15212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6678b6b56a074a27951450ff508f8fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6678b6b56a074a27951450ff508f8fee" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44401 } } }
I20260923 22:19:36.774354 15212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:36.774520 15213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6678b6b56a074a27951450ff508f8fee. Latest consensus state: current_term: 1 leader_uuid: "6678b6b56a074a27951450ff508f8fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6678b6b56a074a27951450ff508f8fee" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44401 } } }
I20260923 22:19:36.774600 15213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:36.774806 15220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:36.775061 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 15140
I20260923 22:19:36.775173 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:19:36.775341 1541 external_mini_cluster.cc:1428] Setting key a1f1efe964c067965d467af23987fb0f
I20260923 22:19:36.775942 15220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:36.777657 15220 catalog_manager.cc:1347] Generated new cluster ID: a1a24c9511594124ab4118ab2edf2cb7
I20260923 22:19:36.777714 15220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:19:36 dist-test-slave-dmw7 krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:36.785843 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:36.775991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.1.129.126:44401)
0923 22:19:36.776186 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:36.776196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:36.776226 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:19:36.776343 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:36.776350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:36.776404 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:36.776542 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:36.777329 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:36.778354 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:36.779197 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:36.779649 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:36.779760 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:36.782001 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:36.782005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:36.782005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:36.782017 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:36.784279 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:36.784640 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:36.784643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:36.784644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:36.784745 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:36.784955 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:36.784956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:36.784956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:36.785203 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:36.785369 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:36.785476 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 22:19:36.785632 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260923 22:19:36.786765 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:19:36.800099 15220 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:36.800937 15220 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:36.805230 15220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6678b6b56a074a27951450ff508f8fee: Generated new TSK 0
I20260923 22:19:36.805428 15220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:19:36.840798 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 15140
2026-09-23T22:19:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1358 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:19:36 dist-test-slave-dmw7 krb5kdc[15235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:19:36 dist-test-slave-dmw7 krb5kdc[15235](info): set up 2 sockets
Sep 23 22:19:36 dist-test-slave-dmw7 krb5kdc[15235](info): commencing operation
krb5kdc: starting...
W20260923 22:19:37.882566 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:19:37 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:19:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:19:37.999341 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35973
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:38.079311 15248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:38.079547 15248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:38.079583 15248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:38.080716 15248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:38.080778 15248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:38.080803 15248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:38.080823 15248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:38.080852 15248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:38.082286 15248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:38.082546 15248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:38.082787 15248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:38.087566 15254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.087535 15253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.087553 15256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:38.087867 15248 server_base.cc:1034] running on GCE node
I20260923 22:19:38.088129 15248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:38.088425 15248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:38.089614 15248 hybrid_clock.cc:648] HybridClock initialized: now 1790201978089605 us; error 35 us; skew 500 ppm
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:38.092178 15248 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:19:38.092600 15248 webserver.cc:460] Webserver started at http://127.1.129.126:35347/ using document root <none> and password file <none>
I20260923 22:19:38.092814 15248 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:38.092877 15248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:38.093015 15248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:38.094079 15248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a977e6b5ca15ff49cf39b163b75ed3ee"
server_key_iv: "3b69b2773ec8334081f95341b7c51fde"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.094419 15248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a977e6b5ca15ff49cf39b163b75ed3ee"
server_key_iv: "3b69b2773ec8334081f95341b7c51fde"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.095778 15248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:19:38.096493 15263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.096704 15248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:38.096802 15248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a977e6b5ca15ff49cf39b163b75ed3ee"
server_key_iv: "3b69b2773ec8334081f95341b7c51fde"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.096877 15248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:38.120093 15248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:38.122509 15248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:38.122715 15248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:38.126528 15248 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:35973
I20260923 22:19:38.126554 15315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:35973 every 8 connection(s)
I20260923 22:19:38.126940 15248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:38.127971 15316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:38.130347 15316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Bootstrap starting.
I20260923 22:19:38.131009 15316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:38.131297 15316 log.cc:824] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:38.132040 15316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: No bootstrap required, opened a new log
I20260923 22:19:38.133772 15316 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:38.133941 15316 raft_consensus.cc:374] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:38.133991 15316 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42e9da9b6aa4b3e8e68e77bacc0fa77, State: Initialized, Role: FOLLOWER
I20260923 22:19:38.134083 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 15248
I20260923 22:19:38.134218 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:19:38.134176 15316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:38.134263 15316 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:38.134311 15316 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:38.134362 15316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:38.134413 1541 external_mini_cluster.cc:1428] Setting key 835dcc9fe03fd563e5139b499d74f9c4
I20260923 22:19:38.135560 15316 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:38.135682 15316 leader_election.cc:302] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42e9da9b6aa4b3e8e68e77bacc0fa77; no voters:
I20260923 22:19:38.135854 15316 leader_election.cc:288] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:19:38.135901 15321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:38.136049 15321 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 1 LEADER]: Becoming Leader. State: Replica: d42e9da9b6aa4b3e8e68e77bacc0fa77, State: Running, Role: LEADER
I20260923 22:19:38.136111 15316 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:38.136196 15321 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:38.137292 15323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42e9da9b6aa4b3e8e68e77bacc0fa77. Latest consensus state: current_term: 1 leader_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } } }
I20260923 22:19:38.137393 15323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:38.137615 15322 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } } }
I20260923 22:19:38.137748 15322 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: This master's current role is: LEADER
W20260923 22:19:38.137892 15336 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:38.137945 15336 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:38.138023 15337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:38.138183 15337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:38.139626 15337 catalog_manager.cc:1347] Generated new cluster ID: 6a2d58d863b44f54b3ff494c1ead16b1
I20260923 22:19:38.139735 15337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:38.146073 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.134973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.1.129.126:35973)
0923 22:19:38.135189 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.135202 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.135247 (+ 45us) server_negotiation.cc:407] Connection header received
0923 22:19:38.135369 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.135377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.135436 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.135592 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.136396 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.137924 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.139013 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.139609 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.139721 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.142415 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.142419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.142420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.142431 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.144499 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.144867 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.144870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.144871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.144991 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.145188 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.145189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.145190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.145501 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.145673 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.145743 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.145881 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260923 22:19:38.167476 15337 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:38.168084 15337 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:38.173647 15337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Generated new TSK 0
I20260923 22:19:38.173867 15337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:19:38.207686 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35973
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:38.289391 15342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:38.289634 15342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:38.289678 15342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:38.290944 15342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:38.291009 15342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:38.291057 15342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:19:38.292515 15342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:38.292774 15342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:38.293018 15342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:38.298003 15348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.297969 15350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.297992 15347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:38.298022 15342 server_base.cc:1034] running on GCE node
I20260923 22:19:38.298463 15342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:38.298766 15342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:38.299928 15342 hybrid_clock.cc:648] HybridClock initialized: now 1790201978299917 us; error 36 us; skew 500 ppm
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:38.306980 15342 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:19:38.307437 15342 webserver.cc:460] Webserver started at http://127.1.129.65:41117/ using document root <none> and password file <none>
I20260923 22:19:38.307667 15342 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:38.307734 15342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:38.307879 15342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:38.308928 15342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "eb4f0724668b48f3ba05ad0b453e2d18"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "b047e11e1c02b62f6e047617d6b886aa"
server_key_iv: "d8fed514c5d652f6d8b0502806c78fc7"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.309311 15342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "eb4f0724668b48f3ba05ad0b453e2d18"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "b047e11e1c02b62f6e047617d6b886aa"
server_key_iv: "d8fed514c5d652f6d8b0502806c78fc7"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.310686 15342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:19:38.311445 15357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.311612 15342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:19:38.311715 15342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "eb4f0724668b48f3ba05ad0b453e2d18"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "b047e11e1c02b62f6e047617d6b886aa"
server_key_iv: "d8fed514c5d652f6d8b0502806c78fc7"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.311786 15342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:38.332348 15342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:38.334306 15342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:38.334474 15342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:38.334733 15342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:38.335048 15342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:38.335098 15342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.335140 15342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:38.335180 15342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.341054 15342 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:40587
I20260923 22:19:38.341105 15470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:40587 every 8 connection(s)
I20260923 22:19:38.341418 15342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:38.343385 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 15342
I20260923 22:19:38.343498 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:19:38.343694 1541 external_mini_cluster.cc:1428] Setting key 9a6dcb3436289c05442e5c3dfc92ac80
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
I20260923 22:19:38.352221 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.342197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:49769 (local address 127.1.129.126:35973)
0923 22:19:38.342273 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.342275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.342492 (+ 217us) server_negotiation.cc:407] Connection header received
0923 22:19:38.343059 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.343060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.343077 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.343134 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.344373 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.345147 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.346550 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.346826 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.346884 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.349349 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.349351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.349351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.349361 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.350820 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.351236 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.351238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.351238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.351294 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.351530 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.351531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.351531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.351650 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.351721 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.352030 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.352129 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 22:19:38.352411 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.342169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:49769)
0923 22:19:38.342359 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.342396 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.342948 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.343212 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.343216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.343959 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.344270 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.344278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.345249 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.345250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.346459 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.346463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.346906 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.346907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.347062 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.347591 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.347601 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.349237 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.350912 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.350920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.350922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.351154 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.351361 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.351362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.351362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.351468 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.351712 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.351718 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.351891 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.352006 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.352169 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260923 22:19:38.352932 15471 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35973
I20260923 22:19:38.353049 15471 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:38.353228 15471 heartbeater.cc:511] Master 127.1.129.126:35973 requested a full tablet report, sending...
I20260923 22:19:38.353791 15280 ts_manager.cc:194] Registered new tserver with Master: eb4f0724668b48f3ba05ad0b453e2d18 (127.1.129.65:40587)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
I20260923 22:19:38.354574 15280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:49769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:19:38.369585 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35973
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:38.450947 15476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:38.451167 15476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:38.451192 15476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:38.452358 15476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:38.452402 15476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:38.452452 15476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:19:38.453890 15476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:38.454134 15476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:38.454365 15476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:38.459503 15481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.459565 15482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.459503 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:38.459925 15476 server_base.cc:1034] running on GCE node
I20260923 22:19:38.460105 15476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:38.460309 15476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:38.461448 15476 hybrid_clock.cc:648] HybridClock initialized: now 1790201978461419 us; error 43 us; skew 500 ppm
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:38.464144 15476 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:19:38.464612 15476 webserver.cc:460] Webserver started at http://127.1.129.66:42851/ using document root <none> and password file <none>
I20260923 22:19:38.464834 15476 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:38.464879 15476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:38.465059 15476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:38.466118 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "302ffce0c88b4efc8f196657f3b1c4a6"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "4594cb1f116e413322815575c073494a"
server_key_iv: "95b71b52e616d52889b3767eb4c5fac3"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.466485 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "302ffce0c88b4efc8f196657f3b1c4a6"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "4594cb1f116e413322815575c073494a"
server_key_iv: "95b71b52e616d52889b3767eb4c5fac3"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.467814 15476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:38.468509 15491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.468715 15476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:19:38.468814 15476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "302ffce0c88b4efc8f196657f3b1c4a6"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "4594cb1f116e413322815575c073494a"
server_key_iv: "95b71b52e616d52889b3767eb4c5fac3"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.468890 15476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:38.478871 15476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:38.481451 15476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:38.481609 15476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:38.481873 15476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:38.482255 15476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:38.482299 15476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.482321 15476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:38.482334 15476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.488204 15476 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:35521
I20260923 22:19:38.488250 15604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:35521 every 8 connection(s)
I20260923 22:19:38.488549 15476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:38.494473 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 15476
I20260923 22:19:38.494603 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:19:38.494796 1541 external_mini_cluster.cc:1428] Setting key 6fbee1353b446b1908ab7f5fea596360
I20260923 22:19:38.497619 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.489222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:45075 (local address 127.1.129.126:35973)
0923 22:19:38.489288 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.489290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.489483 (+ 193us) server_negotiation.cc:407] Connection header received
0923 22:19:38.489918 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.489919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.489929 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.489993 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.490969 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.491460 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.492415 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.492627 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.492685 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.494796 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.494798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.494798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.494806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.496295 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.496698 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.496699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.496700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.496756 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.496988 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.496989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.496989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.497105 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.497171 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.497444 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.497525 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:19:38.497833 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.489202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:45075)
0923 22:19:38.489342 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.489383 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.489842 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.490052 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.490055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.490612 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.490884 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.490890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.491562 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.491562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.492337 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.492339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.492698 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.492699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.492794 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.493218 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.493226 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.494704 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.496380 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.496387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.496389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.496608 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.496818 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.496819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.496819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.496925 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.497174 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.497181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.497317 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.497423 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.497588 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260923 22:19:38.498317 15605 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35973
I20260923 22:19:38.498404 15605 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:38.498585 15605 heartbeater.cc:511] Master 127.1.129.126:35973 requested a full tablet report, sending...
I20260923 22:19:38.499022 15280 ts_manager.cc:194] Registered new tserver with Master: 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
I20260923 22:19:38.499387 15280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:45075
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:19:38.518810 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35973
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:38.599874 15610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:38.600100 15610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:38.600140 15610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:38.601337 15610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:38.601403 15610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:38.601449 15610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:19:38.602994 15610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:38.603250 15610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:38.603470 15610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:19:38.608743 15610 server_base.cc:1034] running on GCE node
W20260923 22:19:38.608697 15615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.608712 15618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.608712 15616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:38.609040 15610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:38.609305 15610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:38.610445 15610 hybrid_clock.cc:648] HybridClock initialized: now 1790201978610432 us; error 29 us; skew 500 ppm
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:38.613088 15610 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:19:38.613524 15610 webserver.cc:460] Webserver started at http://127.1.129.67:35511/ using document root <none> and password file <none>
I20260923 22:19:38.613806 15610 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:38.613873 15610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:38.614007 15610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:38.615155 15610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "e070e94793a042f7b829711adbd616ca"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a205ba370bf2aa8a93343797733e4f5a"
server_key_iv: "c21393bbe548ef7d22e28691637b17a9"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.615511 15610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "e070e94793a042f7b829711adbd616ca"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a205ba370bf2aa8a93343797733e4f5a"
server_key_iv: "c21393bbe548ef7d22e28691637b17a9"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.616808 15610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:38.617568 15625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.617727 15610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:38.617818 15610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "e070e94793a042f7b829711adbd616ca"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a205ba370bf2aa8a93343797733e4f5a"
server_key_iv: "c21393bbe548ef7d22e28691637b17a9"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.617899 15610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:38.631081 15610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:38.633042 15610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:38.633212 15610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:38.633476 15610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:38.633788 15610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:38.633821 15610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.633862 15610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:38.633888 15610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.640257 15610 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:42743
I20260923 22:19:38.640300 15738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:42743 every 8 connection(s)
I20260923 22:19:38.640614 15610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:19:38.643489 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 15610
I20260923 22:19:38.643569 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:19:38.643741 1541 external_mini_cluster.cc:1428] Setting key 882f901d21d880a0b91e1dbd59146570
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:38.650118 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.641296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:57253 (local address 127.1.129.126:35973)
0923 22:19:38.641338 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.641345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.641561 (+ 216us) server_negotiation.cc:407] Connection header received
0923 22:19:38.642192 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.642193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.642205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.642271 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.643281 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.643856 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.645139 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.645322 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.645376 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.647478 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.647479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.647479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.647486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.648836 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.649202 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.649204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.649204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.649260 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.649470 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.649471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.649471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.649582 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.649636 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.649968 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.650051 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:19:38.650279 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.641280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:57253)
0923 22:19:38.641432 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.641475 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.641947 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.642345 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.642349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.642983 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.643191 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.643197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.643964 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.643965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.645060 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.645063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.645394 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.645395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.645494 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.645974 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.645982 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.647397 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.648902 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.648906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.648908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.649114 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.649324 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.649325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.649325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.649417 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.649673 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.649683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.649820 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.649922 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.650059 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260923 22:19:38.650929 15739 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35973
I20260923 22:19:38.651024 15739 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:38.651176 15739 heartbeater.cc:511] Master 127.1.129.126:35973 requested a full tablet report, sending...
I20260923 22:19:38.651594 15280 ts_manager.cc:194] Registered new tserver with Master: e070e94793a042f7b829711adbd616ca (127.1.129.67:42743)
I20260923 22:19:38.651937 15280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:57253
I20260923 22:19:38.655377 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:38.662640 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.656735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.1.129.126:35973)
0923 22:19:38.656800 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.656802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.656900 (+ 98us) server_negotiation.cc:407] Connection header received
0923 22:19:38.656966 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.656967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.656975 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.657013 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.657545 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.658010 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.658847 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.659025 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.659077 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.660297 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.660298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.660299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.660305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.661636 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.661950 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.661951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.661952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.662003 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.662163 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.662163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.662164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.662301 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.662374 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.662475 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.662538 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:19:38.664196 15280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:19:38.665795 15280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:19:38.672513 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.668579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.1.129.66:35521)
0923 22:19:38.668793 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.668798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.668823 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:19:38.668926 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.668928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.668946 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.669042 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:38.669236 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.670026 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.671302 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.672042 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.672125 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.672192 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.672359 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.672420 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260923 22:19:38.672855 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.668433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35521 (local address 127.0.0.1:57800)
0923 22:19:38.668755 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.668767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.668826 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.669052 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.669055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.669057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:38.669140 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.669142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.670133 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.670134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.671188 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.671190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.672123 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.672124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.672216 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.672231 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.672753 (+ 522us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.672791 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":77,"threads_started":1}
I20260923 22:19:38.673036 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.668493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.1.129.67:42743)
0923 22:19:38.668970 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.668975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.669010 (+ 35us) server_negotiation.cc:407] Connection header received
0923 22:19:38.669057 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.669060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.669076 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.669149 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:38.669356 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.670551 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.671711 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.672474 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.672554 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.672612 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.672899 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.672956 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":118,"threads_started":1}
I20260923 22:19:38.673206 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.668763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.1.129.65:40587)
0923 22:19:38.669159 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.669163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.669172 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:38.669211 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.669213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.669227 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.669297 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:38.669491 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.670574 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.671901 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.672721 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.672808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.672867 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.673073 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.673126 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":85,"threads_started":1}
I20260923 22:19:38.672854 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.668385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:42743 (local address 127.0.0.1:33386)
0923 22:19:38.668720 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.668764 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.668828 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.669243 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.669244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.669246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:38.669295 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.669297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.670691 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.670693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.671586 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.671633 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.672563 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.672564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.672631 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.672635 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.672696 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.672767 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":9,"thread_start_us":82,"threads_started":1}
I20260923 22:19:38.673161 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.668618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:40587 (local address 127.0.0.1:41818)
0923 22:19:38.668932 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.668941 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.668977 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.669372 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.669373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.669375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:38.669409 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.669410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.670703 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.670703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.671810 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.671813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.672901 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.672902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.672964 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.672968 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.673053 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.673096 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":39,"threads_started":1}
I20260923 22:19:38.674002 15539 tablet_service.cc:1467] Processing CreateTablet for tablet f28de542d31b45439c4aa22e143713f8 (DEFAULT_TABLE table=test-table [id=e8899b53933d468eb64132cdd7b63922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:38.674111 15673 tablet_service.cc:1467] Processing CreateTablet for tablet f28de542d31b45439c4aa22e143713f8 (DEFAULT_TABLE table=test-table [id=e8899b53933d468eb64132cdd7b63922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:38.674392 15539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28de542d31b45439c4aa22e143713f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:38.674408 15673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28de542d31b45439c4aa22e143713f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:38.674363 15405 tablet_service.cc:1467] Processing CreateTablet for tablet f28de542d31b45439c4aa22e143713f8 (DEFAULT_TABLE table=test-table [id=e8899b53933d468eb64132cdd7b63922]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:38.674705 15405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28de542d31b45439c4aa22e143713f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:38.677134 15753 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: Bootstrap starting.
I20260923 22:19:38.677759 15753 tablet_bootstrap.cc:654] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:38.677862 15754 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Bootstrap starting.
I20260923 22:19:38.678172 15755 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Bootstrap starting.
I20260923 22:19:38.678375 15753 log.cc:824] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:38.678544 15754 tablet_bootstrap.cc:654] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:38.678803 15755 tablet_bootstrap.cc:654] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:38.679131 15754 log.cc:824] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:38.679288 15753 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: No bootstrap required, opened a new log
I20260923 22:19:38.679308 15755 log.cc:824] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:38.679364 15753 ts_tablet_manager.cc:1397] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:38.679867 15754 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: No bootstrap required, opened a new log
I20260923 22:19:38.679940 15754 ts_tablet_manager.cc:1397] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:19:38.680058 15755 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: No bootstrap required, opened a new log
I20260923 22:19:38.680131 15755 ts_tablet_manager.cc:1397] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:38.681315 15754 raft_consensus.cc:348] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.681483 15754 raft_consensus.cc:374] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:38.681391 15753 raft_consensus.cc:348] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.681529 15754 raft_consensus.cc:729] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302ffce0c88b4efc8f196657f3b1c4a6, State: Initialized, Role: FOLLOWER
I20260923 22:19:38.681562 15753 raft_consensus.cc:374] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:38.681608 15753 raft_consensus.cc:729] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e070e94793a042f7b829711adbd616ca, State: Initialized, Role: FOLLOWER
I20260923 22:19:38.681651 15754 consensus_queue.cc:260] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.681720 15753 consensus_queue.cc:260] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.681926 15754 ts_tablet_manager.cc:1428] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:19:38.681984 15753 ts_tablet_manager.cc:1428] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:19:38.682053 15739 heartbeater.cc:503] Master 127.1.129.126:35973 was elected leader, sending a full tablet report...
I20260923 22:19:38.682070 15605 heartbeater.cc:503] Master 127.1.129.126:35973 was elected leader, sending a full tablet report...
I20260923 22:19:38.682111 15755 raft_consensus.cc:348] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.682263 15755 raft_consensus.cc:374] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:38.682297 15755 raft_consensus.cc:729] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4f0724668b48f3ba05ad0b453e2d18, State: Initialized, Role: FOLLOWER
I20260923 22:19:38.682399 15755 consensus_queue.cc:260] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.682698 15471 heartbeater.cc:503] Master 127.1.129.126:35973 was elected leader, sending a full tablet report...
I20260923 22:19:38.682703 15755 ts_tablet_manager.cc:1428] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260923 22:19:38.739079 15606 tablet.cc:2369] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:38.752223 15760 raft_consensus.cc:482] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:38.752367 15760 raft_consensus.cc:504] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.752707 15760 leader_election.cc:288] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521), eb4f0724668b48f3ba05ad0b453e2d18 (127.1.129.65:40587)
I20260923 22:19:38.755662 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.753026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:36899 (local address 127.1.129.66:35521)
0923 22:19:38.753079 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.753081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.753108 (+ 27us) server_negotiation.cc:407] Connection header received
0923 22:19:38.753136 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.753137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.753146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.753201 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:38.753382 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.753863 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.754851 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.755293 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.755375 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.755384 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.755564 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.755604 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 22:19:38.755662 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.752902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35521 (local address 127.1.129.67:36899)
0923 22:19:38.752964 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.752978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.753058 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.753249 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.753250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.753252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:38.753310 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.753311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.753949 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.753950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.754765 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.754767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.755373 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.755374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.755463 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.755466 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.755540 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.755583 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":2}
I20260923 22:19:38.756098 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.753120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:57113 (local address 127.1.129.65:40587)
0923 22:19:38.753177 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.753180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.753273 (+ 93us) server_negotiation.cc:407] Connection header received
0923 22:19:38.753337 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.753338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.753347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.753421 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:38.753575 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.754042 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.755017 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.755675 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.755763 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.755773 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.756013 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.756052 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:19:38.756124 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.753017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:40587 (local address 127.1.129.67:57113)
0923 22:19:38.753201 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.753210 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.753279 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.753424 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.753425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.753427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:38.753501 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.753503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.754125 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.754126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.754921 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.754923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.755832 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.755834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.755930 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.755934 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.755994 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.756029 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":12,"thread_start_us":122,"threads_started":1}
I20260923 22:19:38.756070 15559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28de542d31b45439c4aa22e143713f8" candidate_uuid: "e070e94793a042f7b829711adbd616ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" is_pre_election: true
I20260923 22:19:38.756230 15559 raft_consensus.cc:2454] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e070e94793a042f7b829711adbd616ca in term 0.
I20260923 22:19:38.756464 15628 leader_election.cc:302] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302ffce0c88b4efc8f196657f3b1c4a6, e070e94793a042f7b829711adbd616ca; no voters:
I20260923 22:19:38.756615 15760 raft_consensus.cc:2781] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:38.756680 15760 raft_consensus.cc:482] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:38.756733 15760 raft_consensus.cc:3037] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:38.756837 15425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28de542d31b45439c4aa22e143713f8" candidate_uuid: "e070e94793a042f7b829711adbd616ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" is_pre_election: true
I20260923 22:19:38.756985 15425 raft_consensus.cc:2454] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e070e94793a042f7b829711adbd616ca in term 0.
I20260923 22:19:38.757546 15760 raft_consensus.cc:504] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.757678 15760 leader_election.cc:288] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [CANDIDATE]: Term 1 election: Requested vote from peers 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521), eb4f0724668b48f3ba05ad0b453e2d18 (127.1.129.65:40587)
I20260923 22:19:38.757858 15559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28de542d31b45439c4aa22e143713f8" candidate_uuid: "e070e94793a042f7b829711adbd616ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302ffce0c88b4efc8f196657f3b1c4a6"
I20260923 22:19:38.757891 15425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28de542d31b45439c4aa22e143713f8" candidate_uuid: "e070e94793a042f7b829711adbd616ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb4f0724668b48f3ba05ad0b453e2d18"
I20260923 22:19:38.757936 15559 raft_consensus.cc:3037] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:38.757989 15425 raft_consensus.cc:3037] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:38.758673 15559 raft_consensus.cc:2454] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e070e94793a042f7b829711adbd616ca in term 1.
I20260923 22:19:38.758792 15425 raft_consensus.cc:2454] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e070e94793a042f7b829711adbd616ca in term 1.
I20260923 22:19:38.758827 15628 leader_election.cc:302] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302ffce0c88b4efc8f196657f3b1c4a6, e070e94793a042f7b829711adbd616ca; no voters:
I20260923 22:19:38.758914 15760 raft_consensus.cc:2781] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:38.759075 15760 raft_consensus.cc:686] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 1 LEADER]: Becoming Leader. State: Replica: e070e94793a042f7b829711adbd616ca, State: Running, Role: LEADER
I20260923 22:19:38.759179 15760 consensus_queue.cc:237] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:38.759984 15279 catalog_manager.cc:5495] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e070e94793a042f7b829711adbd616ca (127.1.129.67). New cstate: current_term: 1 leader_uuid: "e070e94793a042f7b829711adbd616ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } health_report { overall_health: UNKNOWN } } }
I20260923 22:19:38.766360 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.764300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.1.129.67:42743)
0923 22:19:38.764343 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.764344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.764351 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:38.764407 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.764407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.764422 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.764479 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:38.764576 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.765015 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.765707 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.765879 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.765931 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.766042 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:38.766130 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:38.766185 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.766277 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.766292 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:19:38.768117 15559 raft_consensus.cc:1261] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Refusing update from remote peer e070e94793a042f7b829711adbd616ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:38.768149 15425 raft_consensus.cc:1261] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Refusing update from remote peer e070e94793a042f7b829711adbd616ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:38.768488 15763 consensus_queue.cc:1035] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:38.768599 15760 consensus_queue.cc:1035] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:38.771047 15768 mvcc.cc:205] Tried to move back new op lower bound from 7332667305031614464 to 7332667304998281216. Current Snapshot: MvccSnapshot[applied={T|T < 7332667305031614464}]
I20260923 22:19:38.771133 15767 mvcc.cc:205] Tried to move back new op lower bound from 7332667305031614464 to 7332667304998281216. Current Snapshot: MvccSnapshot[applied={T|T < 7332667305031614464}]
I20260923 22:19:38.773785 15280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:38.773892 15280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:38.774681 15280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Sending DeleteTablet for 3 replicas of tablet f28de542d31b45439c4aa22e143713f8
I20260923 22:19:38.774988 15673 tablet_service.cc:1514] Processing DeleteTablet for tablet f28de542d31b45439c4aa22e143713f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:38 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:33386
I20260923 22:19:38.775043 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 15342
I20260923 22:19:38.775713 15539 tablet_service.cc:1514] Processing DeleteTablet for tablet f28de542d31b45439c4aa22e143713f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:38 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:57800
I20260923 22:19:38.779520 15776 tablet_replica.cc:330] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: stopping tablet replica
I20260923 22:19:38.779639 15776 raft_consensus.cc:2229] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:19:38.779773 15776 raft_consensus.cc:2258] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:38.779948 15777 tablet_replica.cc:330] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: stopping tablet replica
I20260923 22:19:38.780050 15777 raft_consensus.cc:2229] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260923 22:19:38.780035 15266 connection.cc:535] client connection to 127.1.129.65:40587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 22:19:38.780129 15266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 22:19:38.780169 15777 raft_consensus.cc:2258] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:38.780375 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 15476
I20260923 22:19:38.780592 15776 ts_tablet_manager.cc:1905] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:38.782760 15776 ts_tablet_manager.cc:1918] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:38.782899 15776 log.cc:1196] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/f28de542d31b45439c4aa22e143713f8
I20260923 22:19:38.783296 15776 ts_tablet_manager.cc:1939] T f28de542d31b45439c4aa22e143713f8 P e070e94793a042f7b829711adbd616ca: Deleting consensus metadata
I20260923 22:19:38.783499 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.783332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:40587 (local address 127.0.0.1:41830)
0923 22:19:38.783394 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.783440 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 22:19:38.783636 15266 catalog_manager.cc:4584] TS eb4f0724668b48f3ba05ad0b453e2d18 (127.1.129.65:40587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f28de542d31b45439c4aa22e143713f8: Network error: Client connection negotiation failed: client connection to 127.1.129.65:40587: connect: Connection refused (error 111)
I20260923 22:19:38.783957 15267 catalog_manager.cc:4841] TS e070e94793a042f7b829711adbd616ca (127.1.129.67:42743): tablet f28de542d31b45439c4aa22e143713f8 (table test-table [id=e8899b53933d468eb64132cdd7b63922]) successfully deleted
I20260923 22:19:38.786549 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 15610
I20260923 22:19:38.787794 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.786593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35521 (local address 127.0.0.1:57810)
0923 22:19:38.787741 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.787770 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.66:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1128}
I20260923 22:19:38.791404 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 15248
I20260923 22:19:38.796941 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35973
--webserver_interface=127.1.129.126
--webserver_port=35347
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:38.877610 15779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:38.877818 15779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:38.877840 15779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:38.879034 15779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:38.879076 15779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:38.879089 15779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:38.879101 15779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:38.879113 15779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:38.880595 15779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=35347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:38.880836 15779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:38.881076 15779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:38.889633 15788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.889627 15786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:38.889696 15779 server_base.cc:1034] running on GCE node
W20260923 22:19:38.889698 15785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:38.890246 15779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:38.890560 15779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:38.891698 15779 hybrid_clock.cc:648] HybridClock initialized: now 1790201978891685 us; error 28 us; skew 500 ppm
Sep 23 22:19:38 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:38.894075 15779 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:19:38.894517 15779 webserver.cc:460] Webserver started at http://127.1.129.126:35347/ using document root <none> and password file <none>
I20260923 22:19:38.894799 15779 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:38.894878 15779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:38.896271 15779 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:38.896978 15795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.897174 15779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:38.897279 15779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a977e6b5ca15ff49cf39b163b75ed3ee"
server_key_iv: "3b69b2773ec8334081f95341b7c51fde"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.897702 15779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:38.922518 15779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:38.924917 15779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:38.925112 15779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:38.929147 15779 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:35973
I20260923 22:19:38.929184 15847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:35973 every 8 connection(s)
I20260923 22:19:38.929592 15779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:38.931277 15848 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:19:38.931916 15848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Bootstrap starting.
I20260923 22:19:38.931984 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 15779
I20260923 22:19:38.932252 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:40587
--local_ip_for_outbound_sockets=127.1.129.65
--tserver_master_addrs=127.1.129.126:35973
--webserver_port=41117
--webserver_interface=127.1.129.65
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:38.941075 15848 log.cc:824] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:38.944021 15848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:38.944410 15848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Bootstrap complete.
I20260923 22:19:38.948426 15848 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:38.948752 15848 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42e9da9b6aa4b3e8e68e77bacc0fa77, State: Initialized, Role: FOLLOWER
I20260923 22:19:38.948915 15848 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:38.948982 15848 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:38.949023 15848 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:38.949071 15848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:19:38.950443 15848 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:38.950556 15848 leader_election.cc:302] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42e9da9b6aa4b3e8e68e77bacc0fa77; no voters:
I20260923 22:19:38.950752 15848 leader_election.cc:288] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 22:19:38.950827 15852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:19:38.950995 15852 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 2 LEADER]: Becoming Leader. State: Replica: d42e9da9b6aa4b3e8e68e77bacc0fa77, State: Running, Role: LEADER
I20260923 22:19:38.951009 15848 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:38.952279 15852 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:38.952831 15854 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42e9da9b6aa4b3e8e68e77bacc0fa77. Latest consensus state: current_term: 2 leader_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } } }
I20260923 22:19:38.952873 15853 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } } }
I20260923 22:19:38.953140 15854 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:38.953151 15853 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:38.953509 15866 catalog_manager.cc:1259] Loaded cluster ID: 6a2d58d863b44f54b3ff494c1ead16b1
I20260923 22:19:38.953555 15866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: loading cluster ID for follower catalog manager: success
W20260923 22:19:38.953758 15866 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260923 22:19:38.953799 15866 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260923 22:19:38.953857 15868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:38.954197 15868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:38.954270 15868 catalog_manager.cc:1259] Loaded cluster ID: 6a2d58d863b44f54b3ff494c1ead16b1
I20260923 22:19:38.954298 15868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:19:38.954453 15868 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 22:19:38.959731 15868 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 22:19:38.962252 15868 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:38.962451 15868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Loaded TSK: 0
I20260923 22:19:38.962710 15868 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:19:39.025517 15850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:39.025727 15850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:39.025751 15850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:39.026976 15850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:39.027020 15850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:39.027047 15850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:19:39.028491 15850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:40587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=41117
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:39.028738 15850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:39.028981 15850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:39.034924 15878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.034996 15850 server_base.cc:1034] running on GCE node
W20260923 22:19:39.034897 15876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:39.035153 15875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.035382 15850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:39.035650 15850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:39.036796 15850 hybrid_clock.cc:648] HybridClock initialized: now 1790201979036779 us; error 33 us; skew 500 ppm
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:39.039290 15850 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:19:39.039786 15850 webserver.cc:460] Webserver started at http://127.1.129.65:41117/ using document root <none> and password file <none>
I20260923 22:19:39.040025 15850 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:39.040082 15850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:39.041447 15850 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:39.042133 15885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:39.042281 15850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:39.042362 15850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "eb4f0724668b48f3ba05ad0b453e2d18"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "b047e11e1c02b62f6e047617d6b886aa"
server_key_iv: "d8fed514c5d652f6d8b0502806c78fc7"
server_key_version: "encryptionkey@0"
I20260923 22:19:39.042766 15850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:39.052449 15850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:39.054589 15850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:39.054790 15850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:39.055032 15850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:39.055526 15892 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:19:39.056990 15850 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:19:39.057040 15850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 22:19:39.057089 15850 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:19:39.057726 15850 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:19:39.057770 15850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:39.057816 15892 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Bootstrap starting.
I20260923 22:19:39.065318 15850 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:40587
I20260923 22:19:39.065449 15999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:40587 every 8 connection(s)
I20260923 22:19:39.065932 15850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:39.068764 15892 log.cc:824] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:39.071352 15892 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:39.071645 15892 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Bootstrap complete.
I20260923 22:19:39.072435 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 15850
I20260923 22:19:39.072695 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:35521
--local_ip_for_outbound_sockets=127.1.129.66
--tserver_master_addrs=127.1.129.126:35973
--webserver_port=42851
--webserver_interface=127.1.129.66
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:39.072719 15892 ts_tablet_manager.cc:1397] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260923 22:19:39.074452 15892 raft_consensus.cc:348] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:39.074856 15892 raft_consensus.cc:729] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4f0724668b48f3ba05ad0b453e2d18, State: Initialized, Role: FOLLOWER
I20260923 22:19:39.074990 15892 consensus_queue.cc:260] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:39.075285 15892 ts_tablet_manager.cc:1428] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:39.082046 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.066914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:44721 (local address 127.1.129.126:35973)
0923 22:19:39.067112 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.067122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.069177 (+ 2055us) server_negotiation.cc:407] Connection header received
0923 22:19:39.069959 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.069966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.070027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.070180 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:39.071045 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.072440 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.074054 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.074679 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.074757 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.077836 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:39.077840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:39.077841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:39.077854 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:39.080033 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.080583 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.080587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.080587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.080693 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.080996 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.080997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.080997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.081277 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:39.081355 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.081711 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.081850 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260923 22:19:39.082103 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.066781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:44721)
0923 22:19:39.069012 (+ 2231us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.069047 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.069715 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.070235 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:39.070239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:39.070603 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:39.070922 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.070928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.072556 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.072557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.073952 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.073955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.074872 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.074872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.074996 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.075696 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:39.075706 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:39.077703 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:39.080134 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:39.080142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:39.080144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:39.080489 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:39.080771 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:39.080772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:39.080773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:39.080925 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:39.081349 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:39.081355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:39.081531 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 22:19:39.081659 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 22:19:39.081858 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2192,"thread_start_us":98,"threads_started":1}
I20260923 22:19:39.083173 16000 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35973
I20260923 22:19:39.083281 16000 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:39.083489 16000 heartbeater.cc:511] Master 127.1.129.126:35973 requested a full tablet report, sending...
I20260923 22:19:39.084297 15812 ts_manager.cc:194] Registered new tserver with Master: eb4f0724668b48f3ba05ad0b453e2d18 (127.1.129.65:40587)
I20260923 22:19:39.085443 15812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:44721
I20260923 22:19:39.086400 16000 heartbeater.cc:503] Master 127.1.129.126:35973 was elected leader, sending a full tablet report...
I20260923 22:19:39.094552 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.085108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.1.129.65:40587)
0923 22:19:39.091044 (+ 5936us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.091049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.091058 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:39.091091 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.091093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.091108 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.091191 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:39.091352 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.092460 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.093404 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.094139 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.094221 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.094280 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.094429 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.094477 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5903,"thread_start_us":80,"threads_started":1}
I20260923 22:19:39.094647 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.084878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:40587 (local address 127.0.0.1:41832)
0923 22:19:39.090791 (+ 5913us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.090837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.090907 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.091212 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:39.091216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:39.091219 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:39.091277 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.091279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.092538 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.092538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.093326 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.093328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.094225 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.094226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.094283 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.094293 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 22:19:39.094361 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 22:19:39.094399 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5876,"thread_start_us":85,"threads_started":1}
I20260923 22:19:39.095249 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet f28de542d31b45439c4aa22e143713f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:38 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:41832
I20260923 22:19:39.095875 16009 tablet_replica.cc:330] stopping tablet replica
I20260923 22:19:39.095983 16009 raft_consensus.cc:2229] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:39.096069 16009 raft_consensus.cc:2258] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:39.096380 16009 ts_tablet_manager.cc:1905] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:39.098538 16009 ts_tablet_manager.cc:1918] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:39.098603 16009 log.cc:1196] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/f28de542d31b45439c4aa22e143713f8
I20260923 22:19:39.098853 16009 ts_tablet_manager.cc:1939] T f28de542d31b45439c4aa22e143713f8 P eb4f0724668b48f3ba05ad0b453e2d18: Deleting consensus metadata
I20260923 22:19:39.099349 15798 catalog_manager.cc:4841] TS eb4f0724668b48f3ba05ad0b453e2d18 (127.1.129.65:40587): tablet f28de542d31b45439c4aa22e143713f8 (table test-table [id=e8899b53933d468eb64132cdd7b63922]) successfully deleted
W20260923 22:19:39.162863 16005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:39.163115 16005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:39.163154 16005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:39.164318 16005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:39.164389 16005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:39.164433 16005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:19:39.165894 16005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:35521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=42851
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:39.166163 16005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:39.166396 16005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:39.171450 16014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:39.171463 16015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:39.171450 16017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.171785 16005 server_base.cc:1034] running on GCE node
I20260923 22:19:39.172029 16005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:39.172294 16005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:39.173470 16005 hybrid_clock.cc:648] HybridClock initialized: now 1790201979173466 us; error 37 us; skew 500 ppm
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:39.176055 16005 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:19:39.176501 16005 webserver.cc:460] Webserver started at http://127.1.129.66:42851/ using document root <none> and password file <none>
I20260923 22:19:39.176733 16005 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:39.176817 16005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:39.178263 16005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:39.179032 16024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:39.179206 16005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:39.179306 16005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "302ffce0c88b4efc8f196657f3b1c4a6"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "4594cb1f116e413322815575c073494a"
server_key_iv: "95b71b52e616d52889b3767eb4c5fac3"
server_key_version: "encryptionkey@0"
I20260923 22:19:39.179673 16005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:39.200423 16005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:39.202543 16005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:39.202783 16005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:39.203050 16005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:39.203562 16031 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:19:39.205027 16005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:19:39.205097 16005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 22:19:39.205143 16005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:19:39.205861 16005 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:19:39.205925 16005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:39.206012 16031 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Bootstrap starting.
I20260923 22:19:39.207821 16031 log.cc:824] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:39.210992 16031 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:39.211304 16031 tablet_bootstrap.cc:492] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Bootstrap complete.
I20260923 22:19:39.211475 16031 ts_tablet_manager.cc:1397] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260923 22:19:39.212626 16005 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:35521
I20260923 22:19:39.212697 16139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:35521 every 8 connection(s)
I20260923 22:19:39.213001 16005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:19:39.212965 16031 raft_consensus.cc:348] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:39.213241 16031 raft_consensus.cc:729] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302ffce0c88b4efc8f196657f3b1c4a6, State: Initialized, Role: FOLLOWER
I20260923 22:19:39.213608 16031 consensus_queue.cc:260] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } }
I20260923 22:19:39.213866 16031 ts_tablet_manager.cc:1428] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 22:19:39.218950 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 16005
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:39.219244 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:42743
--local_ip_for_outbound_sockets=127.1.129.67
--tserver_master_addrs=127.1.129.126:35973
--webserver_port=35511
--webserver_interface=127.1.129.67
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:39.222909 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.213694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:41631 (local address 127.1.129.126:35973)
0923 22:19:39.213744 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.213746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.213989 (+ 243us) server_negotiation.cc:407] Connection header received
0923 22:19:39.214182 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.214184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.214200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.214252 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:39.214920 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.215438 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.216474 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.216700 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.216791 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.219772 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:39.219774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:39.219774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:39.219783 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:39.221334 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.221876 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.221877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.221877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.221932 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.222235 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.222235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.222236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.222343 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:39.222393 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.222764 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.222844 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:19:39.223240 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.213686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:41631)
0923 22:19:39.213854 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.213887 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.214087 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.214318 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:39.214322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:39.214598 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:39.214817 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.214822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.215563 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.215564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.216349 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.216351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.216805 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.216806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.216940 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.217440 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:39.217448 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:39.219666 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:39.221425 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:39.221436 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:39.221439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:39.221792 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:39.222004 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:39.222005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:39.222005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:39.222170 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:39.222407 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:39.222412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:39.222591 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 22:19:39.222751 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 22:19:39.222955 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260923 22:19:39.223822 16140 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35973
I20260923 22:19:39.223937 16140 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:39.224156 16140 heartbeater.cc:511] Master 127.1.129.126:35973 requested a full tablet report, sending...
I20260923 22:19:39.224949 15812 ts_manager.cc:194] Registered new tserver with Master: 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
I20260923 22:19:39.225620 15812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:41631
I20260923 22:19:39.226544 16140 heartbeater.cc:503] Master 127.1.129.126:35973 was elected leader, sending a full tablet report...
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.66@KRBTEST.COM
I20260923 22:19:39.233673 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.225404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.1.129.66:35521)
0923 22:19:39.225603 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.225609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.225882 (+ 273us) server_negotiation.cc:407] Connection header received
0923 22:19:39.225917 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.225918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.225930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.225999 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:39.226422 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.227425 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.228119 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.228443 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.228531 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.230948 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:39.230955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:39.230955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:39.230961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:39.232438 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.232867 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.232869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.232869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.232920 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.233104 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.233105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.233105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.233294 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:39.233357 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.233521 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.233593 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":113,"threads_started":1}
I20260923 22:19:39.233690 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.225313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35521 (local address 127.0.0.1:57826)
0923 22:19:39.225764 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.225775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.225819 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.226014 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:39.226015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:39.226307 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:39.226340 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.226342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.227509 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.227510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.228020 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.228021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.228534 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.228535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.228599 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.229167 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:39.229171 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:39.230856 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:39.232516 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:39.232524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:39.232526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:39.232787 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:39.232982 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:39.232983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:39.232983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:39.233048 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:39.233357 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:39.233363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:39.233415 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 22:19:39.233488 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 22:19:39.233590 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433}
I20260923 22:19:39.234417 16074 tablet_service.cc:1514] Processing DeleteTablet for tablet f28de542d31b45439c4aa22e143713f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:38 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:57826
I20260923 22:19:39.235113 16146 tablet_replica.cc:330] stopping tablet replica
I20260923 22:19:39.235231 16146 raft_consensus.cc:2229] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:39.235322 16146 raft_consensus.cc:2258] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:39.235623 16146 ts_tablet_manager.cc:1905] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:39.237608 16146 ts_tablet_manager.cc:1918] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:39.237670 16146 log.cc:1196] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/f28de542d31b45439c4aa22e143713f8
I20260923 22:19:39.237901 16146 ts_tablet_manager.cc:1939] T f28de542d31b45439c4aa22e143713f8 P 302ffce0c88b4efc8f196657f3b1c4a6: Deleting consensus metadata
I20260923 22:19:39.238374 15798 catalog_manager.cc:4841] TS 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521): tablet f28de542d31b45439c4aa22e143713f8 (table test-table [id=e8899b53933d468eb64132cdd7b63922]) successfully deleted
W20260923 22:19:39.304427 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:39.304663 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:39.304703 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:39.305840 16144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:39.305903 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:39.305949 16144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:19:39.307507 16144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:42743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=35511
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:39.307781 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:39.308017 16144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:39.313087 16151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:39.313068 16152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.313215 16144 server_base.cc:1034] running on GCE node
W20260923 22:19:39.313068 16154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.313544 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:39.313831 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:39.314989 16144 hybrid_clock.cc:648] HybridClock initialized: now 1790201979314984 us; error 39 us; skew 500 ppm
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:39.317565 16144 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:19:39.318056 16144 webserver.cc:460] Webserver started at http://127.1.129.67:35511/ using document root <none> and password file <none>
I20260923 22:19:39.318248 16144 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:39.318300 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:39.319713 16144 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:39.320290 16161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:39.320484 16144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:39.320590 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "e070e94793a042f7b829711adbd616ca"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a205ba370bf2aa8a93343797733e4f5a"
server_key_iv: "c21393bbe548ef7d22e28691637b17a9"
server_key_version: "encryptionkey@0"
I20260923 22:19:39.320928 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:39.348673 16144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:39.351246 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:39.351364 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:39.351609 16144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:39.352015 16144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:39.352068 16144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:39.352094 16144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:39.352106 16144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:39.358174 16144 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:42743
I20260923 22:19:39.358255 16274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:42743 every 8 connection(s)
I20260923 22:19:39.358610 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:39.365128 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 16144
I20260923 22:19:39.368342 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.359405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:43821 (local address 127.1.129.126:35973)
0923 22:19:39.359455 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.359457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.359705 (+ 248us) server_negotiation.cc:407] Connection header received
0923 22:19:39.360221 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.360222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.360232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.360285 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:39.361232 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.361718 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.362738 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.362918 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.362972 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.365139 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:39.365140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:39.365140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:39.365147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:39.366933 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.367323 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.367324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.367325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.367391 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.367667 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.367668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.367668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.367785 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:39.367862 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.368168 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.368244 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:19:39.368563 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.359395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:43821)
0923 22:19:39.359556 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.359593 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.360114 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.360345 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:39.360349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:39.360932 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:39.361146 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.361151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.361799 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.361800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.362615 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.362617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.362991 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.362992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.363097 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.363596 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:39.363604 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:39.365026 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:39.367007 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:39.367014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:39.367016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:39.367249 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:39.367458 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:39.367459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:39.367459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:39.367598 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:39.367857 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:39.367865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:39.368015 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 22:19:39.368136 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 22:19:39.368311 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260923 22:19:39.369056 16275 heartbeater.cc:347] Connected to a master server at 127.1.129.126:35973
I20260923 22:19:39.369184 16275 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:39.369401 16275 heartbeater.cc:511] Master 127.1.129.126:35973 requested a full tablet report, sending...
I20260923 22:19:39.369843 15812 ts_manager.cc:194] Registered new tserver with Master: e070e94793a042f7b829711adbd616ca (127.1.129.67:42743)
I20260923 22:19:39.370357 15812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:43821
I20260923 22:19:39.372694 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.365588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.1.129.126:35973)
0923 22:19:39.365700 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.365702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.365737 (+ 35us) server_negotiation.cc:407] Connection header received
0923 22:19:39.365820 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.365821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.365834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.365911 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:39.366560 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.367270 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.368251 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.368544 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.368607 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.370262 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:39.370263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:39.370263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:39.370269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:39.371783 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.372101 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.372102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.372102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.372152 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:39.372271 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:39.372271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:39.372271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:39.372361 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:39.372470 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.372529 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.372593 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260923 22:19:39.373433 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:39.376384 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.373726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.1.129.126:35973)
0923 22:19:39.373767 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.373769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.373929 (+ 160us) server_negotiation.cc:407] Connection header received
0923 22:19:39.374004 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.374005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.374041 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.374113 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:39.374269 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.374821 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.375572 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.375789 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.375872 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.376054 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:39.376158 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:39.376244 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.376281 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.376305 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:19:39.376739 15812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:19:39.378316 15812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:19:39.380416 16074 tablet_service.cc:1467] Processing CreateTablet for tablet b9acde9e856b436088e304a847577dfc (DEFAULT_TABLE table=test-table [id=821cf5e23cbe47bd9448f05530a0f337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:39.380630 16074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9acde9e856b436088e304a847577dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:39.382230 16031 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: Bootstrap starting.
I20260923 22:19:39.382397 15916 tablet_service.cc:1467] Processing CreateTablet for tablet b9acde9e856b436088e304a847577dfc (DEFAULT_TABLE table=test-table [id=821cf5e23cbe47bd9448f05530a0f337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:39.382535 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9acde9e856b436088e304a847577dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:39.382689 16031 tablet_bootstrap.cc:654] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:39.384075 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.380112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:42743 (local address 127.0.0.1:33406)
0923 22:19:39.380169 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.380183 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.380221 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.380576 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:39.380578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:39.380580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:39.380651 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.380653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.381901 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.381902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.382926 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.382930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.383866 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.383867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.383916 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.383919 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:19:39.383954 (+ 35us) client_negotiation.cc:240] Negotiation successful
0923 22:19:39.383988 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 22:19:39.384073 15892 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Bootstrap starting.
I20260923 22:19:39.384101 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.380198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.1.129.67:42743)
0923 22:19:39.380407 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.380411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.380421 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:19:39.380462 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.380464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.380480 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.380564 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:39.380763 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.381798 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.383000 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.383792 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.383870 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.383934 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.383968 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.384013 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":123,"threads_started":1}
I20260923 22:19:39.384409 15892 tablet_bootstrap.cc:654] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:39.384433 16031 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: No bootstrap required, opened a new log
I20260923 22:19:39.384473 16031 ts_tablet_manager.cc:1397] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:19:39.384800 16031 raft_consensus.cc:348] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.384927 16031 raft_consensus.cc:374] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:39.384982 16031 raft_consensus.cc:729] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302ffce0c88b4efc8f196657f3b1c4a6, State: Initialized, Role: FOLLOWER
I20260923 22:19:39.384986 15892 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: No bootstrap required, opened a new log
I20260923 22:19:39.385025 15892 ts_tablet_manager.cc:1397] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:39.385035 16031 consensus_queue.cc:260] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.385128 16031 ts_tablet_manager.cc:1428] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:39.385007 16209 tablet_service.cc:1467] Processing CreateTablet for tablet b9acde9e856b436088e304a847577dfc (DEFAULT_TABLE table=test-table [id=821cf5e23cbe47bd9448f05530a0f337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:39.385124 15892 raft_consensus.cc:348] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.385206 15892 raft_consensus.cc:374] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:39.385257 15892 raft_consensus.cc:729] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4f0724668b48f3ba05ad0b453e2d18, State: Initialized, Role: FOLLOWER
I20260923 22:19:39.385322 15892 consensus_queue.cc:260] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.385344 16209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9acde9e856b436088e304a847577dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:39.385416 15892 ts_tablet_manager.cc:1428] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:39.388206 16284 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Bootstrap starting.
I20260923 22:19:39.388885 16284 tablet_bootstrap.cc:654] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:39.389319 16284 log.cc:824] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:39.389963 16284 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: No bootstrap required, opened a new log
I20260923 22:19:39.390017 16284 ts_tablet_manager.cc:1397] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:39.391734 16284 raft_consensus.cc:348] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.391917 16284 raft_consensus.cc:374] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:39.391952 16284 raft_consensus.cc:729] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e070e94793a042f7b829711adbd616ca, State: Initialized, Role: FOLLOWER
I20260923 22:19:39.392074 16284 consensus_queue.cc:260] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.392381 16284 ts_tablet_manager.cc:1428] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:19:39.392387 16275 heartbeater.cc:503] Master 127.1.129.126:35973 was elected leader, sending a full tablet report...
W20260923 22:19:39.464118 16141 tablet.cc:2369] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:19:39.567646 16001 tablet.cc:2369] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:39.573621 16006 raft_consensus.cc:482] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:39.573807 16006 raft_consensus.cc:504] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.574198 16006 leader_election.cc:288] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e070e94793a042f7b829711adbd616ca (127.1.129.67:42743), 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
I20260923 22:19:39.578132 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.574479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:45437 (local address 127.1.129.67:42743)
0923 22:19:39.574574 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.574576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.574584 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:19:39.574619 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.574620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.574681 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.574742 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:39.575217 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.575882 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.577180 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.577755 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.577841 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.577851 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.578020 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.578062 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 22:19:39.578157 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.574481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:42387 (local address 127.1.129.66:35521)
0923 22:19:39.574529 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.574531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.574698 (+ 167us) server_negotiation.cc:407] Connection header received
0923 22:19:39.574739 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.574740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.574765 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.574831 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:39.574983 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.575714 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.577180 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.577794 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.577863 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.577935 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.578018 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.578059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:19:39.578155 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.574405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:42743 (local address 127.1.129.65:45437)
0923 22:19:39.574454 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.574466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.574511 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.574764 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:39.574765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:39.574767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:39.575142 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.575144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.575947 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.575948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.577071 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.577074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.577862 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.577863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.577942 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.577946 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:39.578005 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 22:19:39.578057 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":32}
I20260923 22:19:39.578330 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.574406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35521 (local address 127.1.129.65:42387)
0923 22:19:39.574597 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.574607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.574690 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.574847 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:39.574848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:39.574849 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:39.574913 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.574914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.575796 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.575797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.577071 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:39.577074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.577889 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.577890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:39.577946 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.577948 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:19:39.578005 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 22:19:39.578053 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":88,"thread_start_us":60,"threads_started":1}
I20260923 22:19:39.578521 16229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9acde9e856b436088e304a847577dfc" candidate_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e070e94793a042f7b829711adbd616ca" is_pre_election: true
I20260923 22:19:39.578727 16229 raft_consensus.cc:2454] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4f0724668b48f3ba05ad0b453e2d18 in term 0.
I20260923 22:19:39.578783 16094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9acde9e856b436088e304a847577dfc" candidate_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" is_pre_election: true
I20260923 22:19:39.578928 16094 raft_consensus.cc:2454] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4f0724668b48f3ba05ad0b453e2d18 in term 0.
I20260923 22:19:39.578986 15889 leader_election.cc:302] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e070e94793a042f7b829711adbd616ca, eb4f0724668b48f3ba05ad0b453e2d18; no voters:
I20260923 22:19:39.579140 16006 raft_consensus.cc:2781] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:39.579231 16006 raft_consensus.cc:482] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:39.579269 16006 raft_consensus.cc:3037] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:39.579977 16006 raft_consensus.cc:504] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.580128 16006 leader_election.cc:288] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 1 election: Requested vote from peers e070e94793a042f7b829711adbd616ca (127.1.129.67:42743), 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
I20260923 22:19:39.580327 16229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9acde9e856b436088e304a847577dfc" candidate_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e070e94793a042f7b829711adbd616ca"
I20260923 22:19:39.580405 16229 raft_consensus.cc:3037] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:39.580384 16094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9acde9e856b436088e304a847577dfc" candidate_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302ffce0c88b4efc8f196657f3b1c4a6"
I20260923 22:19:39.580493 16094 raft_consensus.cc:3037] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:39.581120 16229 raft_consensus.cc:2454] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4f0724668b48f3ba05ad0b453e2d18 in term 1.
I20260923 22:19:39.581246 16094 raft_consensus.cc:2454] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4f0724668b48f3ba05ad0b453e2d18 in term 1.
I20260923 22:19:39.581349 15889 leader_election.cc:302] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e070e94793a042f7b829711adbd616ca, eb4f0724668b48f3ba05ad0b453e2d18; no voters:
I20260923 22:19:39.581436 16006 raft_consensus.cc:2781] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:39.581575 16006 raft_consensus.cc:686] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 LEADER]: Becoming Leader. State: Replica: eb4f0724668b48f3ba05ad0b453e2d18, State: Running, Role: LEADER
I20260923 22:19:39.581671 16006 consensus_queue.cc:237] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.582571 15811 catalog_manager.cc:5495] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb4f0724668b48f3ba05ad0b453e2d18 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } health_report { overall_health: UNKNOWN } } }
W20260923 22:19:39.609223 16276 tablet.cc:2369] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:39.611194 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.608786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.1.129.65:40587)
0923 22:19:39.608913 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:39.608915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:39.608924 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:39.608972 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:39.608973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:39.608992 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:39.609061 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:39.609235 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.609789 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.610500 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:39.610708 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:39.610757 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:39.610887 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:39.610980 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:39.611031 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:39.611070 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 22:19:39.611088 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260923 22:19:39.612618 16094 raft_consensus.cc:1261] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Refusing update from remote peer eb4f0724668b48f3ba05ad0b453e2d18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:39.612627 16229 raft_consensus.cc:1261] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 1 FOLLOWER]: Refusing update from remote peer eb4f0724668b48f3ba05ad0b453e2d18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:39.612938 16006 consensus_queue.cc:1035] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:39.613076 16288 consensus_queue.cc:1035] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:39.615700 16292 mvcc.cc:205] Tried to move back new op lower bound from 7332667308490575872 to 7332667308367204352. Current Snapshot: MvccSnapshot[applied={T|T < 7332667308490575872 or (T in {7332667308490575872})}]
I20260923 22:19:39.617393 15812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:39.617491 15812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:39.618207 15812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Sending DeleteTablet for 3 replicas of tablet b9acde9e856b436088e304a847577dfc
I20260923 22:19:39.618470 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet b9acde9e856b436088e304a847577dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:39 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:41832
I20260923 22:19:39.618507 16074 tablet_service.cc:1514] Processing DeleteTablet for tablet b9acde9e856b436088e304a847577dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:39 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:57826
I20260923 22:19:39.618547 16209 tablet_service.cc:1514] Processing DeleteTablet for tablet b9acde9e856b436088e304a847577dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:39 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:33406
I20260923 22:19:39.618600 16146 tablet_replica.cc:330] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: stopping tablet replica
I20260923 22:19:39.618672 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 15850
I20260923 22:19:39.618681 16146 raft_consensus.cc:2229] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:39.618731 16146 raft_consensus.cc:2258] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:39.618993 16302 tablet_replica.cc:330] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: stopping tablet replica
I20260923 22:19:39.619346 16302 raft_consensus.cc:2229] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:39.619508 16302 raft_consensus.cc:2258] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:39.620018 16302 ts_tablet_manager.cc:1905] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:39.620455 16146 ts_tablet_manager.cc:1905] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:39.621964 16146 ts_tablet_manager.cc:1918] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:39.621961 16302 ts_tablet_manager.cc:1918] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:39.622026 16146 log.cc:1196] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/b9acde9e856b436088e304a847577dfc
I20260923 22:19:39.622036 16302 log.cc:1196] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/b9acde9e856b436088e304a847577dfc
I20260923 22:19:39.622274 16146 ts_tablet_manager.cc:1939] T b9acde9e856b436088e304a847577dfc P 302ffce0c88b4efc8f196657f3b1c4a6: Deleting consensus metadata
I20260923 22:19:39.622299 16302 ts_tablet_manager.cc:1939] T b9acde9e856b436088e304a847577dfc P e070e94793a042f7b829711adbd616ca: Deleting consensus metadata
I20260923 22:19:39.622591 15798 catalog_manager.cc:4841] TS 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521): tablet b9acde9e856b436088e304a847577dfc (table test-table [id=821cf5e23cbe47bd9448f05530a0f337]) successfully deleted
I20260923 22:19:39.622808 15799 catalog_manager.cc:4841] TS e070e94793a042f7b829711adbd616ca (127.1.129.67:42743): tablet b9acde9e856b436088e304a847577dfc (table test-table [id=821cf5e23cbe47bd9448f05530a0f337]) successfully deleted
W20260923 22:19:39.625017 15798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:40587 (error 108)
I20260923 22:19:39.625131 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 16005
I20260923 22:19:39.626528 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.626386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:40587 (local address 127.0.0.1:41848)
0923 22:19:39.626450 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.626497 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 22:19:39.626685 15798 catalog_manager.cc:4584] TS eb4f0724668b48f3ba05ad0b453e2d18 (127.1.129.65:40587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9acde9e856b436088e304a847577dfc: Network error: Client connection negotiation failed: client connection to 127.1.129.65:40587: connect: Connection refused (error 111)
I20260923 22:19:39.629912 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 16144
I20260923 22:19:39.634735 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 15779
I20260923 22:19:39.639704 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35973
--webserver_interface=127.1.129.126
--webserver_port=35347
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:39.720396 16304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:39.720600 16304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:39.720621 16304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:39.721747 16304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:39.721823 16304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:39.721850 16304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:39.721864 16304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:39.721876 16304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:39.723683 16304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=35347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:39.723941 16304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:39.724180 16304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:39.729111 16310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:39.729154 16312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.729324 16304 server_base.cc:1034] running on GCE node
W20260923 22:19:39.729142 16309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.729631 16304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:39.729939 16304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:39.731102 16304 hybrid_clock.cc:648] HybridClock initialized: now 1790201979731083 us; error 30 us; skew 500 ppm
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:39.733536 16304 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:19:39.733964 16304 webserver.cc:460] Webserver started at http://127.1.129.126:35347/ using document root <none> and password file <none>
I20260923 22:19:39.734200 16304 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:39.734267 16304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:39.735631 16304 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:39.736275 16319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:39.736449 16304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:39.736547 16304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a977e6b5ca15ff49cf39b163b75ed3ee"
server_key_iv: "3b69b2773ec8334081f95341b7c51fde"
server_key_version: "encryptionkey@0"
I20260923 22:19:39.736912 16304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:39.750102 16304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:39.751950 16304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:39.752105 16304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:39.756450 16304 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:35973
I20260923 22:19:39.756518 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:35973 every 8 connection(s)
I20260923 22:19:39.756947 16304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:39.758445 16372 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:19:39.759143 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Bootstrap starting.
I20260923 22:19:39.764580 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 16304
I20260923 22:19:39.764951 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:40587
--local_ip_for_outbound_sockets=127.1.129.65
--tserver_master_addrs=127.1.129.126:35973
--webserver_port=41117
--webserver_interface=127.1.129.65
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:39.767812 16372 log.cc:824] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:39.770054 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:39.770399 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77: Bootstrap complete.
I20260923 22:19:39.772668 16372 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:39.772887 16372 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42e9da9b6aa4b3e8e68e77bacc0fa77, State: Initialized, Role: FOLLOWER
I20260923 22:19:39.773048 16372 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:39.773120 16372 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:39.773185 16372 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:39.773231 16372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 2 FOLLOWER]: Advancing to term 3
I20260923 22:19:39.774243 16372 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:39.774328 16372 leader_election.cc:302] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42e9da9b6aa4b3e8e68e77bacc0fa77; no voters:
I20260923 22:19:39.774534 16372 leader_election.cc:288] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 22:19:39.774614 16376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 22:19:39.774756 16376 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [term 3 LEADER]: Becoming Leader. State: Replica: d42e9da9b6aa4b3e8e68e77bacc0fa77, State: Running, Role: LEADER
I20260923 22:19:39.774833 16372 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:39.774876 16376 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } }
I20260923 22:19:39.775225 16377 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } } }
I20260923 22:19:39.775310 16377 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:39.775513 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42e9da9b6aa4b3e8e68e77bacc0fa77. Latest consensus state: current_term: 3 leader_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e9da9b6aa4b3e8e68e77bacc0fa77" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 35973 } } }
I20260923 22:19:39.775625 16383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:39.776419 16383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:39.776572 16383 catalog_manager.cc:1259] Loaded cluster ID: 6a2d58d863b44f54b3ff494c1ead16b1
I20260923 22:19:39.775609 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e9da9b6aa4b3e8e68e77bacc0fa77 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:39.776731 16383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 22:19:39.777194 16383 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f73d2bf19ad google::LogMessage::Fail() at ??:0
@ 0x7f73d2bf488a google::LogMessage::SendToLog() at ??:0
@ 0x7f73d2bf14de google::LogMessage::Flush() at ??:0
@ 0x7f73d2bf2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f73dab160f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f73dab1f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f73d4abf0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f73d4ab4954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73d59186db start_thread at ??:0
@ 0x7f73d16ef61f clone at ??:0
W20260923 22:19:39.863183 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:39.863412 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:39.863452 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:39.864622 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:39.864688 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:39.864735 16374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:19:39.866204 16374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:40587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=41117
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:39.866465 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:39.866760 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:39.871840 16408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:39.871963 16411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.872004 16374 server_base.cc:1034] running on GCE node
W20260923 22:19:39.871832 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:39.872298 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:39.872560 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:39.873718 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790201979873701 us; error 41 us; skew 500 ppm
Sep 23 22:19:39 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:39.876305 16374 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:19:39.876775 16374 webserver.cc:460] Webserver started at http://127.1.129.65:41117/ using document root <none> and password file <none>
I20260923 22:19:39.877022 16374 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:39.877090 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:39.878448 16374 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:39.879215 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:39.879429 16374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:39.879526 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "eb4f0724668b48f3ba05ad0b453e2d18"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "b047e11e1c02b62f6e047617d6b886aa"
server_key_iv: "d8fed514c5d652f6d8b0502806c78fc7"
server_key_version: "encryptionkey@0"
I20260923 22:19:39.879966 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:39.913938 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:39.916424 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:39.916581 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:39.916851 16374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:39.917331 16425 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:19:39.919262 16374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:19:39.919323 16374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 22:19:39.919368 16374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:19:39.920001 16374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:19:39.920061 16374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:39.920125 16425 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Bootstrap starting.
I20260923 22:19:39.927155 16374 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:40587
I20260923 22:19:39.927323 16532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:40587 every 8 connection(s)
I20260923 22:19:39.927577 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:39.930778 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 16374
I20260923 22:19:39.931041 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:35521
--local_ip_for_outbound_sockets=127.1.129.66
--tserver_master_addrs=127.1.129.126:35973
--webserver_port=42851
--webserver_interface=127.1.129.66
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:39.931077 16425 log.cc:824] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:39.933722 16425 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:39.934033 16425 tablet_bootstrap.cc:492] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Bootstrap complete.
I20260923 22:19:39.935277 16425 ts_tablet_manager.cc:1397] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.010s
I20260923 22:19:39.936834 16425 raft_consensus.cc:348] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.937170 16425 raft_consensus.cc:729] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4f0724668b48f3ba05ad0b453e2d18, State: Initialized, Role: FOLLOWER
I20260923 22:19:39.937335 16425 consensus_queue.cc:260] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:39.937634 16425 ts_tablet_manager.cc:1428] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260923 22:19:39.994208 16535 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:19:39.928332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:46249)
0923 22:19:39.928543 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.928580 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:39.929019 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:39.993998 (+ 64979us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":122,"threads_started":1}
W20260923 22:19:39.994500 16533 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:35973 (0 consecutive failures): Network error: Failed to ping master at 127.1.129.126:35973: Client connection negotiation failed: client connection to 127.1.129.126:35973: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 22:19:39.994890 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.994773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:54185)
0923 22:19:39.994826 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.994871 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:19:39.995280 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:39.995192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:35611)
0923 22:19:39.995242 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:39.995265 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 22:19:39.995400 16533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:19:40.017597 16537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:40.017824 16537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:40.017849 16537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:40.019064 16537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:40.019105 16537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:40.019133 16537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:19:40.020592 16537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:35521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=42851
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:40.020848 16537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:40.021090 16537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:40.026065 16544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:40.026078 16543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:40.026078 16546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:40.026294 16537 server_base.cc:1034] running on GCE node
I20260923 22:19:40.026515 16537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:40.026829 16537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:40.027974 16537 hybrid_clock.cc:648] HybridClock initialized: now 1790201980027961 us; error 27 us; skew 500 ppm
Sep 23 22:19:40 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:40.030387 16537 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:19:40.030880 16537 webserver.cc:460] Webserver started at http://127.1.129.66:42851/ using document root <none> and password file <none>
I20260923 22:19:40.031103 16537 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:40.031150 16537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:40.032472 16537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 22:19:40.033023 16553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:40.033198 16537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:40.033294 16537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "302ffce0c88b4efc8f196657f3b1c4a6"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "4594cb1f116e413322815575c073494a"
server_key_iv: "95b71b52e616d52889b3767eb4c5fac3"
server_key_version: "encryptionkey@0"
I20260923 22:19:40.033640 16537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:40.049304 16537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:40.051348 16537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:40.051517 16537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:40.051764 16537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:40.052073 16537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:40.052124 16537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:40.052160 16537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:40.052187 16537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:40.058336 16537 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:35521
I20260923 22:19:40.058383 16666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:35521 every 8 connection(s)
I20260923 22:19:40.058734 16537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:19:40.059796 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:40.059367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:40193)
0923 22:19:40.059543 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:40.059653 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
W20260923 22:19:40.060225 16667 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:35973 (0 consecutive failures): Network error: Failed to ping master at 127.1.129.126:35973: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
I20260923 22:19:40.060527 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:40.060441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:60655)
0923 22:19:40.060485 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:40.060510 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 22:19:40.060783 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:40.060735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:34475)
0923 22:19:40.060755 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:40.060771 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260923 22:19:40.060923 16667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:19:40.066170 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 16537
I20260923 22:19:40.066383 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:42743
--local_ip_for_outbound_sockets=127.1.129.67
--tserver_master_addrs=127.1.129.126:35973
--webserver_port=35511
--webserver_interface=127.1.129.67
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:40.145530 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:40.145728 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:40.145751 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:40.146829 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:40.146867 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:40.146893 16670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:19:40.148226 16670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:42743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=35511
--tserver_master_addrs=127.1.129.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:40.148392 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:40.148559 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:40.153486 16678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:40.153512 16675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:40.153688 16670 server_base.cc:1034] running on GCE node
W20260923 22:19:40.153699 16676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:40.153910 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:40.154186 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:40.155349 16670 hybrid_clock.cc:648] HybridClock initialized: now 1790201980155331 us; error 41 us; skew 500 ppm
Sep 23 22:19:40 dist-test-slave-dmw7 krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:40.157747 16670 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:19:40.158167 16670 webserver.cc:460] Webserver started at http://127.1.129.67:35511/ using document root <none> and password file <none>
I20260923 22:19:40.158389 16670 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:40.158459 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:40.159857 16670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:40.160431 16685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:40.160609 16670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:40.160702 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "e070e94793a042f7b829711adbd616ca"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-dmw7"
server_key: "a205ba370bf2aa8a93343797733e4f5a"
server_key_iv: "c21393bbe548ef7d22e28691637b17a9"
server_key_version: "encryptionkey@0"
I20260923 22:19:40.161029 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:40.170261 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:40.172132 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:40.172287 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:40.172508 16670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:40.172798 16670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:40.172850 16670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:40.172889 16670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:40.172917 16670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:40.178639 16670 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:42743
I20260923 22:19:40.178707 16798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:42743 every 8 connection(s)
I20260923 22:19:40.179054 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:19:40.180069 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:40.179714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:50641)
0923 22:19:40.179864 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:40.179932 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
W20260923 22:19:40.180496 16799 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:35973 (0 consecutive failures): Network error: Failed to ping master at 127.1.129.126:35973: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
I20260923 22:19:40.180778 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:40.180685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:49099)
0923 22:19:40.180729 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:40.180757 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 22:19:40.181021 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 16670
I20260923 22:19:40.181084 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:40.180992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:48773)
0923 22:19:40.181037 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:40.181062 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:19:40.181201 16799 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:19:40.995994 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:40.995743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:44153)
0923 22:19:40.995884 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:40.995932 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260923 22:19:41.061507 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:41.061282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:52943)
0923 22:19:41.061410 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:41.061451 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260923 22:19:41.181702 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:41.181514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:36469)
0923 22:19:41.181619 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:41.181649 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260923 22:19:41.197001 16806 raft_consensus.cc:482] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:41.197162 16806 raft_consensus.cc:504] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:41.197494 16806 leader_election.cc:288] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e070e94793a042f7b829711adbd616ca (127.1.129.67:42743), 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
Sep 23 22:19:41 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.66@KRBTEST.COM
Sep 23 22:19:41 dist-test-slave-dmw7 krb5kdc[15235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201979, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.67@KRBTEST.COM
I20260923 22:19:41.208007 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:41.197710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35521 (local address 127.1.129.65:57951)
0923 22:19:41.197871 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:41.197883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:41.197937 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:41.198703 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:41.198706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:41.199071 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:41.199246 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:41.199251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:41.200328 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:41.200329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:41.201201 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:41.201203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:41.201780 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:41.201781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:41.201872 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:41.202490 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:41.202497 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:41.204001 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:41.206153 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:41.206159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:41.206161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:41.206512 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:41.206814 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:41.206814 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:41.206815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:41.206967 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:41.207412 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:41.207417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:41.207569 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 22:19:41.207710 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 22:19:41.207885 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":129,"thread_start_us":59,"threads_started":1}
I20260923 22:19:41.208026 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:41.197839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:57951 (local address 127.1.129.66:35521)
0923 22:19:41.197985 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:41.197995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:41.198022 (+ 27us) server_negotiation.cc:407] Connection header received
0923 22:19:41.198453 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:41.198455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:41.198493 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:41.198601 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:41.199381 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:41.200205 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:41.201296 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:41.201685 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:41.201801 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:41.204099 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:41.204103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:41.204103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:41.204112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:41.206061 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:41.206598 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:41.206600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:41.206601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:41.206747 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:41.207040 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:41.207041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:41.207041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:41.207337 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:41.207422 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:41.207784 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 22:19:41.207913 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260923 22:19:41.208052 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:41.197710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:42743 (local address 127.1.129.65:43223)
0923 22:19:41.197775 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:41.197788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:41.197857 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:41.198703 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:41.198706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:41.199069 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:41.199246 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:41.199251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:41.200295 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:41.200296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:41.201202 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:41.201205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:41.201762 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:41.201763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:41.201871 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:41.202443 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:41.202450 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:41.204413 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:41.206365 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:41.206366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:41.206366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:41.206615 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:41.206879 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:41.206880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:41.206880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:41.206967 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:41.207412 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:41.207417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:41.207569 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 22:19:41.207689 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 22:19:41.207939 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":110}
I20260923 22:19:41.208027 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:41.197849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:43223 (local address 127.1.129.67:42743)
0923 22:19:41.197992 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:41.198001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:41.198039 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:19:41.198476 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:41.198478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:41.198513 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:41.198685 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:41.199364 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:41.200198 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:41.201284 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:41.201678 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:41.201787 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:41.204506 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:41.204509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:41.204509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:41.204518 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:41.206281 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:41.206716 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:41.206718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:41.206718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:41.206817 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:41.207049 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:41.207050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:41.207050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:41.207337 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:41.207422 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:41.207784 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 22:19:41.207913 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":51,"threads_started":1}
W20260923 22:19:41.209132 16421 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
W20260923 22:19:41.209434 16422 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e070e94793a042f7b829711adbd616ca (127.1.129.67:42743): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
I20260923 22:19:41.209502 16422 leader_election.cc:302] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb4f0724668b48f3ba05ad0b453e2d18; no voters: 302ffce0c88b4efc8f196657f3b1c4a6, e070e94793a042f7b829711adbd616ca
I20260923 22:19:41.209615 16806 raft_consensus.cc:2726] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:41.996816 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:41.996546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:45715)
0923 22:19:41.996710 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:41.996768 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260923 22:19:42.062247 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.062024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:39601)
0923 22:19:42.062163 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.062206 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260923 22:19:42.182350 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.182146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:60469)
0923 22:19:42.182270 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.182308 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260923 22:19:42.785012 16813 raft_consensus.cc:482] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:42.785120 16813 raft_consensus.cc:504] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:42.785315 16813 leader_election.cc:288] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e070e94793a042f7b829711adbd616ca (127.1.129.67:42743), 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
W20260923 22:19:42.785751 16421 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
W20260923 22:19:42.785835 16422 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e070e94793a042f7b829711adbd616ca (127.1.129.67:42743): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
I20260923 22:19:42.785913 16422 leader_election.cc:302] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb4f0724668b48f3ba05ad0b453e2d18; no voters: 302ffce0c88b4efc8f196657f3b1c4a6, e070e94793a042f7b829711adbd616ca
I20260923 22:19:42.786001 16813 raft_consensus.cc:2726] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:42.997624 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.997394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:52137)
0923 22:19:42.997520 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.997563 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260923 22:19:43.063061 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.062831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:47567)
0923 22:19:43.062950 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.063010 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260923 22:19:43.183004 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.182815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:57075)
0923 22:19:43.182931 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.182965 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260923 22:19:43.998399 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.998150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:45371)
0923 22:19:43.998314 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.998353 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260923 22:19:44.063690 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.063487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:42375)
0923 22:19:44.063608 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.063646 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260923 22:19:44.183637 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.183433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:58937)
0923 22:19:44.183552 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.183592 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260923 22:19:44.999209 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.998969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:60001)
0923 22:19:44.999122 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.999161 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260923 22:19:45.064394 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.064148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:48307)
0923 22:19:45.064307 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:45.064351 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260923 22:19:45.184419 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.184187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:33745)
0923 22:19:45.184332 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:45.184376 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260923 22:19:45.474879 16823 raft_consensus.cc:482] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:45.474964 16823 raft_consensus.cc:504] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:45.475168 16823 leader_election.cc:288] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e070e94793a042f7b829711adbd616ca (127.1.129.67:42743), 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
W20260923 22:19:45.475584 16421 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
W20260923 22:19:45.475664 16422 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e070e94793a042f7b829711adbd616ca (127.1.129.67:42743): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
I20260923 22:19:45.475746 16422 leader_election.cc:302] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb4f0724668b48f3ba05ad0b453e2d18; no voters: 302ffce0c88b4efc8f196657f3b1c4a6, e070e94793a042f7b829711adbd616ca
I20260923 22:19:45.475837 16823 raft_consensus.cc:2726] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:45.999881 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.999685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:44239)
0923 22:19:45.999801 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:45.999834 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260923 22:19:46.065176 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:46.064971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:44775)
0923 22:19:46.065100 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:46.065135 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260923 22:19:46.185079 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:46.184858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:35509)
0923 22:19:46.184996 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:46.185038 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260923 22:19:47.000644 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:47.000404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:47709)
0923 22:19:47.000551 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:47.000598 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260923 22:19:47.065804 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:47.065594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:51257)
0923 22:19:47.065721 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:47.065762 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260923 22:19:47.185747 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:47.185538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:50715)
0923 22:19:47.185652 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:47.185704 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260923 22:19:48.001407 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:48.001181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:50693)
0923 22:19:48.001320 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:48.001361 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260923 22:19:48.066470 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:48.066258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:35407)
0923 22:19:48.066386 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:48.066425 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260923 22:19:48.186524 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:48.186285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:50881)
0923 22:19:48.186437 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:48.186480 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260923 22:19:49.002118 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:49.001887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:50197)
0923 22:19:49.002033 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:49.002073 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260923 22:19:49.050077 16834 raft_consensus.cc:482] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:49.050223 16834 raft_consensus.cc:504] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:49.050464 16834 leader_election.cc:288] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e070e94793a042f7b829711adbd616ca (127.1.129.67:42743), 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
W20260923 22:19:49.050922 16422 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e070e94793a042f7b829711adbd616ca (127.1.129.67:42743): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
W20260923 22:19:49.050997 16421 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
I20260923 22:19:49.051069 16421 leader_election.cc:302] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb4f0724668b48f3ba05ad0b453e2d18; no voters: 302ffce0c88b4efc8f196657f3b1c4a6, e070e94793a042f7b829711adbd616ca
I20260923 22:19:49.051170 16834 raft_consensus.cc:2726] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:49.067225 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:49.066983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:34383)
0923 22:19:49.067115 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:49.067163 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260923 22:19:49.187316 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:49.187048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:54297)
0923 22:19:49.187216 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:49.187261 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260923 22:19:50.002816 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:50.002558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:41719)
0923 22:19:50.002713 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:50.002769 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":56,"threads_started":1}
I20260923 22:19:50.067888 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:50.067658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:42315)
0923 22:19:50.067809 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:50.067844 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260923 22:19:50.188028 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:50.187816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:37861)
0923 22:19:50.187943 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:50.187984 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260923 22:19:51.003575 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:51.003328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:50045)
0923 22:19:51.003481 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:51.003531 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260923 22:19:51.068599 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:51.068355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:35257)
0923 22:19:51.068508 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:51.068546 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260923 22:19:51.188714 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:51.188507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:53235)
0923 22:19:51.188626 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:51.188664 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260923 22:19:51.910028 16843 raft_consensus.cc:482] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:51.910156 16843 raft_consensus.cc:504] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e070e94793a042f7b829711adbd616ca" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42743 } } peers { permanent_uuid: "eb4f0724668b48f3ba05ad0b453e2d18" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 40587 } } peers { permanent_uuid: "302ffce0c88b4efc8f196657f3b1c4a6" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35521 } }
I20260923 22:19:51.910393 16843 leader_election.cc:288] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e070e94793a042f7b829711adbd616ca (127.1.129.67:42743), 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521)
W20260923 22:19:51.910756 16422 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e070e94793a042f7b829711adbd616ca (127.1.129.67:42743): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
W20260923 22:19:51.910841 16421 leader_election.cc:341] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 302ffce0c88b4efc8f196657f3b1c4a6 (127.1.129.66:35521): Not found: Tablet not found: b9acde9e856b436088e304a847577dfc
I20260923 22:19:51.910929 16421 leader_election.cc:302] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb4f0724668b48f3ba05ad0b453e2d18; no voters: 302ffce0c88b4efc8f196657f3b1c4a6, e070e94793a042f7b829711adbd616ca
I20260923 22:19:51.911005 16843 raft_consensus.cc:2726] T b9acde9e856b436088e304a847577dfc P eb4f0724668b48f3ba05ad0b453e2d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:52.004379 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:52.004052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:41101)
0923 22:19:52.004227 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:52.004309 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20260923 22:19:52.069283 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:52.069069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:46127)
0923 22:19:52.069199 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:52.069235 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260923 22:19:52.189491 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:52.189276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:51855)
0923 22:19:52.189398 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:52.189441 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260923 22:19:53.005419 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:53.004989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:49345)
0923 22:19:53.005307 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:53.005362 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":68,"threads_started":1}
I20260923 22:19:53.070061 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:53.069825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:44361)
0923 22:19:53.069974 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:53.070016 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260923 22:19:53.190200 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:53.189988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:34949)
0923 22:19:53.190117 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:53.190153 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260923 22:19:54.006431 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:54.006182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:32853)
0923 22:19:54.006335 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:54.006385 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260923 22:19:54.070997 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:54.070732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:52469)
0923 22:19:54.070909 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:54.070950 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260923 22:19:54.190934 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:54.190722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:45259)
0923 22:19:54.190851 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:54.190890 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260923 22:19:55.007268 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:55.007053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.65:55209)
0923 22:19:55.007181 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:55.007218 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260923 22:19:55.071761 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:55.071540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.66:56579)
0923 22:19:55.071680 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:55.071720 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":58,"threads_started":1}
I20260923 22:19:55.189289 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 16374
I20260923 22:19:55.191673 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:55.191479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:35973 (local address 127.1.129.67:43747)
0923 22:19:55.191592 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:55.191632 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260923 22:19:55.194206 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 16537
I20260923 22:19:55.198400 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 16670
2026-09-23T22:19:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18359 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:19:55 dist-test-slave-dmw7 krb5kdc[16858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:19:55 dist-test-slave-dmw7 krb5kdc[16858](info): set up 2 sockets
Sep 23 22:19:55 dist-test-slave-dmw7 krb5kdc[16858](info): commencing operation
krb5kdc: starting...
W20260923 22:19:56.230571 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:19:56.343478 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37263
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:56.422747 16871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:56.422964 16871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:56.422988 16871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:56.424103 16871 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 22:19:56.424139 16871 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 22:19:56.424152 16871 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 22:19:56.424165 16871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:56.424177 16871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:56.424190 16871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:56.424202 16871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:56.424214 16871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:56.425623 16871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:37263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:56.425829 16871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:56.426004 16871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:19:56.431238 16871 server_base.cc:1034] running on GCE node
W20260923 22:19:56.431435 16877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:56.431358 16876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:56.431353 16879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:56.431984 16871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:56.432276 16871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:56.433430 16871 hybrid_clock.cc:648] HybridClock initialized: now 1790201996433412 us; error 30 us; skew 500 ppm
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:56.436175 16871 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:19:56.436604 16871 webserver.cc:460] Webserver started at http://127.1.129.126:44583/ using document root <none> and password file <none>
I20260923 22:19:56.436864 16871 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:56.436913 16871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:56.437063 16871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:56.438242 16871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "28c91cd2dcde4673bc3ca362449c693a"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "2469a915d0ab0c30da03d303ea3711eb"
server_key_iv: "ff40bf162f21508ee4cd1d859918e3ec"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.438619 16871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "28c91cd2dcde4673bc3ca362449c693a"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "2469a915d0ab0c30da03d303ea3711eb"
server_key_iv: "ff40bf162f21508ee4cd1d859918e3ec"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.440083 16871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:56.440887 16886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.441115 16871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:56.441232 16871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "28c91cd2dcde4673bc3ca362449c693a"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "2469a915d0ab0c30da03d303ea3711eb"
server_key_iv: "ff40bf162f21508ee4cd1d859918e3ec"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.441318 16871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:56.457938 16871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:56.460024 16871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:56.460223 16871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:56.464353 16871 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:37263
I20260923 22:19:56.464421 16938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:37263 every 8 connection(s)
I20260923 22:19:56.464826 16871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:56.465683 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:56.467985 16939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Bootstrap starting.
I20260923 22:19:56.468061 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 16871
I20260923 22:19:56.468201 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:19:56.468405 1541 external_mini_cluster.cc:1428] Setting key 0e43833ffa81261af029f929c01d3bc1
I20260923 22:19:56.468713 16939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:56.469058 16939 log.cc:824] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:56.469712 16939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: No bootstrap required, opened a new log
I20260923 22:19:56.471984 16939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:19:56.472153 16939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:56.472208 16939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c91cd2dcde4673bc3ca362449c693a, State: Initialized, Role: FOLLOWER
I20260923 22:19:56.472342 16939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:19:56.472411 16939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:56.472438 16939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:56.472481 16939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:56.473052 16939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:19:56.473140 16939 leader_election.cc:302] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c91cd2dcde4673bc3ca362449c693a; no voters:
I20260923 22:19:56.473335 16939 leader_election.cc:288] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:19:56.473354 16944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:56.473573 16944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 1 LEADER]: Becoming Leader. State: Replica: 28c91cd2dcde4673bc3ca362449c693a, State: Running, Role: LEADER
I20260923 22:19:56.473605 16939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:56.474363 16944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:19:56.475046 16945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c91cd2dcde4673bc3ca362449c693a. Latest consensus state: current_term: 1 leader_uuid: "28c91cd2dcde4673bc3ca362449c693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } } }
I20260923 22:19:56.475310 16946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28c91cd2dcde4673bc3ca362449c693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } } }
I20260923 22:19:56.475153 16945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:56.475378 16946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: This master's current role is: LEADER
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
W20260923 22:19:56.475890 16958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:56.475956 16958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:56.476044 16959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:56.476248 16959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:56.477878 16959 catalog_manager.cc:1347] Generated new cluster ID: 607fd89636f748058cb687d00841f282
I20260923 22:19:56.477929 16959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:19:56.479827 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.469034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.1.129.126:37263)
0923 22:19:56.469256 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:56.469264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:56.469290 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:19:56.469699 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:56.469705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:56.469808 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:56.469933 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:56.470747 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.471640 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.472688 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.473080 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.473223 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:56.476184 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:56.476188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:56.476189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:56.476201 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:56.478169 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.478591 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.478594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.478594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.478724 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.478925 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.478926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.478926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.479225 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:56.479389 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:56.479466 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 22:19:56.479607 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260923 22:19:56.491155 16959 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:56.491681 16959 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:56.509723 16959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Generated new TSK 0
I20260923 22:19:56.509974 16959 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:19:56.542392 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37263
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:56.625134 16965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:56.625344 16965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:56.625367 16965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:56.626451 16965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:56.626484 16965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:56.626509 16965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:19:56.627916 16965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:56.628099 16965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:56.628265 16965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:56.633272 16970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:56.633291 16971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:56.633327 16973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:56.633335 16965 server_base.cc:1034] running on GCE node
I20260923 22:19:56.633731 16965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:56.633975 16965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:56.635135 16965 hybrid_clock.cc:648] HybridClock initialized: now 1790201996635122 us; error 29 us; skew 500 ppm
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:56.637761 16965 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:19:56.638192 16965 webserver.cc:460] Webserver started at http://127.1.129.65:40915/ using document root <none> and password file <none>
I20260923 22:19:56.638453 16965 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:56.638517 16965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:56.638702 16965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:56.639741 16965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "f061a104f7194879bd12edbcfd22e8d9"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "43bf9ae3e77c8e03699cd2d083765c27"
server_key_iv: "109ad55990af4ca1af635a6fb1ea5106"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.640100 16965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "f061a104f7194879bd12edbcfd22e8d9"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "43bf9ae3e77c8e03699cd2d083765c27"
server_key_iv: "109ad55990af4ca1af635a6fb1ea5106"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.641409 16965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:19:56.642103 16980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.642258 16965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:56.642354 16965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "f061a104f7194879bd12edbcfd22e8d9"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "43bf9ae3e77c8e03699cd2d083765c27"
server_key_iv: "109ad55990af4ca1af635a6fb1ea5106"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.642442 16965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:56.660688 16965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:56.663094 16965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:56.663268 16965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:56.663515 16965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:56.663832 16965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:56.663885 16965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.663920 16965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:56.663947 16965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.669664 16965 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39141
I20260923 22:19:56.669731 17093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39141 every 8 connection(s)
I20260923 22:19:56.670017 16965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:56.677919 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 16965
I20260923 22:19:56.678043 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:19:56.678273 1541 external_mini_cluster.cc:1428] Setting key 6995b0c9cd56a42943b6f8faa95c760d
I20260923 22:19:56.679639 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.670730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:56807 (local address 127.1.129.126:37263)
0923 22:19:56.670808 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:56.670811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:56.671079 (+ 268us) server_negotiation.cc:407] Connection header received
0923 22:19:56.671537 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:56.671538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:56.671550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:56.671605 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:56.672530 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.673044 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.674189 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.674415 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.674487 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:56.676708 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:56.676709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:56.676710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:56.676716 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:56.678081 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.678613 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.678614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.678615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.678696 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.678974 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.678975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.678975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.679096 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:56.679180 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:56.679458 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 22:19:56.679547 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 22:19:56.679877 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.670737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:56807)
0923 22:19:56.670949 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:56.670986 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:56.671448 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:56.671659 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:56.671663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:56.672248 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:56.672445 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:56.672449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.673128 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.673128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:56.674101 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:56.674103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.674496 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.674497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:56.674615 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:56.675190 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:56.675199 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:56.676622 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:56.678184 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:56.678193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:56.678196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:56.678526 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:56.678781 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:56.678782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:56.678783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:56.678902 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:56.679170 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:56.679177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:56.679325 (+ 148us) client_negotiation.cc:769] Sending connection context
0923 22:19:56.679438 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 22:19:56.679604 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260923 22:19:56.680331 17094 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37263
I20260923 22:19:56.680439 17094 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:56.680650 17094 heartbeater.cc:511] Master 127.1.129.126:37263 requested a full tablet report, sending...
I20260923 22:19:56.681207 16903 ts_manager.cc:194] Registered new tserver with Master: f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141)
I20260923 22:19:56.681826 16903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:56807
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:19:56.705651 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37263
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:56.786657 17099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:56.786871 17099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:56.786895 17099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:56.788059 17099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:56.788105 17099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:56.788139 17099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:19:56.789568 17099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:56.789819 17099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:56.790073 17099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:56.795269 17107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:56.795322 17104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:56.795286 17105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:56.795517 17099 server_base.cc:1034] running on GCE node
I20260923 22:19:56.795718 17099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:56.795953 17099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:56.797096 17099 hybrid_clock.cc:648] HybridClock initialized: now 1790201996797080 us; error 32 us; skew 500 ppm
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:56.799676 17099 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:19:56.800218 17099 webserver.cc:460] Webserver started at http://127.1.129.66:42413/ using document root <none> and password file <none>
I20260923 22:19:56.800460 17099 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:56.800526 17099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:56.800659 17099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:56.801680 17099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "e22365fbe1a54772bbd55b848b9c12d1"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "f37397922359bf8f3ba70a156858285e"
server_key_iv: "cfcb012b27a20a9ef3fdd0467196bb08"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.802044 17099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "e22365fbe1a54772bbd55b848b9c12d1"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "f37397922359bf8f3ba70a156858285e"
server_key_iv: "cfcb012b27a20a9ef3fdd0467196bb08"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.803433 17099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 22:19:56.804139 17114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.804314 17099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:56.804411 17099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "e22365fbe1a54772bbd55b848b9c12d1"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "f37397922359bf8f3ba70a156858285e"
server_key_iv: "cfcb012b27a20a9ef3fdd0467196bb08"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.804497 17099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:56.823390 17099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:56.825282 17099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:56.825450 17099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:56.825692 17099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:56.825991 17099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:56.826045 17099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.826088 17099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:56.826113 17099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.832114 17099 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:34519
I20260923 22:19:56.832155 17227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:34519 every 8 connection(s)
I20260923 22:19:56.832468 17099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:56.840751 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 17099
I20260923 22:19:56.840870 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:19:56.840951 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.833186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:54025 (local address 127.1.129.126:37263)
0923 22:19:56.833272 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:56.833274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:56.833431 (+ 157us) server_negotiation.cc:407] Connection header received
0923 22:19:56.833925 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:56.833926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:56.833936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:56.833990 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:56.834907 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.835391 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.836345 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.836516 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.836563 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:56.838566 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:56.838568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:56.838568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:56.838574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:56.839764 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.840121 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.840121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.840122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.840169 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.840372 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.840372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.840372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.840469 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:56.840517 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:56.840758 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 22:19:56.840840 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 22:19:56.841078 1541 external_mini_cluster.cc:1428] Setting key d959bdb8097395a5118d203f42720274
I20260923 22:19:56.841137 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.833143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:54025)
0923 22:19:56.833314 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:56.833351 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:56.833842 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:56.834046 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:56.834049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:56.834615 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:56.834832 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:56.834837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.835471 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.835472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:56.836271 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:56.836273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.836584 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.836585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:56.836681 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:56.837153 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:56.837160 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:56.838487 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:56.839834 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:56.839840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:56.839841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:56.840057 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:56.840226 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:56.840227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:56.840227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:56.840320 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:56.840523 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:56.840526 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:56.840633 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 22:19:56.840711 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 22:19:56.840880 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260923 22:19:56.841560 17228 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37263
I20260923 22:19:56.841677 17228 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:56.841892 17228 heartbeater.cc:511] Master 127.1.129.126:37263 requested a full tablet report, sending...
I20260923 22:19:56.842303 16902 ts_manager.cc:194] Registered new tserver with Master: e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519)
I20260923 22:19:56.842746 16902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:54025
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:19:56.865482 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37263
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:56.944049 17233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:56.944273 17233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:56.944311 17233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:56.945457 17233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:56.945513 17233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:56.945541 17233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:19:56.947113 17233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:19:56.947289 17233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:56.947453 17233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:56.952435 17239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:56.952474 17238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:56.952474 17241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:56.952507 17233 server_base.cc:1034] running on GCE node
I20260923 22:19:56.952857 17233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:56.953116 17233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:56.954280 17233 hybrid_clock.cc:648] HybridClock initialized: now 1790201996954257 us; error 34 us; skew 500 ppm
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:56.956663 17233 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:19:56.957108 17233 webserver.cc:460] Webserver started at http://127.1.129.67:42265/ using document root <none> and password file <none>
I20260923 22:19:56.957326 17233 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:56.957394 17233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:56.957542 17233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:56.958554 17233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "849e7c2d71d0463d98bdea2c2557e928"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "45321c8f1680527c266f95dccaf831c4"
server_key_iv: "18a505ba3a9ec543cbcaa981249b385c"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.958973 17233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "849e7c2d71d0463d98bdea2c2557e928"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "45321c8f1680527c266f95dccaf831c4"
server_key_iv: "18a505ba3a9ec543cbcaa981249b385c"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.960275 17233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:56.960949 17248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.961241 17233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:56.961336 17233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "849e7c2d71d0463d98bdea2c2557e928"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "45321c8f1680527c266f95dccaf831c4"
server_key_iv: "18a505ba3a9ec543cbcaa981249b385c"
server_key_version: "encryptionkey@0"
I20260923 22:19:56.961410 17233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:56.967703 17233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:56.969810 17233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:56.969964 17233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:56.970180 17233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:56.970482 17233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:56.970536 17233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.970577 17233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:56.970605 17233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:56.976804 17233 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44599
I20260923 22:19:56.976848 17361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44599 every 8 connection(s)
I20260923 22:19:56.977144 17233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:19:56.980033 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 17233
I20260923 22:19:56.980129 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:19:56.980304 1541 external_mini_cluster.cc:1428] Setting key 6f1836a53caa78560c45bff6e0d21bee
Sep 23 22:19:56 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201996, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:19:56.986827 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.977808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:53483 (local address 127.1.129.126:37263)
0923 22:19:56.977865 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:56.977866 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:56.978037 (+ 171us) server_negotiation.cc:407] Connection header received
0923 22:19:56.978552 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:56.978554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:56.978563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:56.978671 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:56.979558 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.980266 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.981493 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.981682 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.981736 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:56.983974 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:56.983975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:56.983975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:56.983982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:56.985423 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.985880 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.985882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.985882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.985945 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.986190 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.986190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.986190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.986305 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:56.986360 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:56.986674 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 22:19:56.986759 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:19:56.987169 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.977781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:53483)
0923 22:19:56.977920 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:56.977959 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:56.978465 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:56.978745 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:56.978749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:56.979293 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:56.979477 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:56.979482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.980369 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.980370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:56.981425 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:56.981427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.981752 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.981753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:56.981931 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:56.982372 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:56.982379 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:56.983868 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:56.985541 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:56.985547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:56.985549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:56.985786 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:56.986018 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:56.986019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:56.986019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:56.986119 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:56.986386 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:56.986392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:56.986513 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 22:19:56.986605 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 22:19:56.986993 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260923 22:19:56.987737 17362 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37263
I20260923 22:19:56.987849 17362 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:56.988029 17362 heartbeater.cc:511] Master 127.1.129.126:37263 requested a full tablet report, sending...
I20260923 22:19:56.988405 16902 ts_manager.cc:194] Registered new tserver with Master: 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599)
I20260923 22:19:56.988740 16902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:53483
I20260923 22:19:56.991807 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:56.999243 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.993110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.1.129.126:37263)
0923 22:19:56.993180 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:56.993182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:56.993271 (+ 89us) server_negotiation.cc:407] Connection header received
0923 22:19:56.993348 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:56.993349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:56.993357 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:56.993429 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:56.994023 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.994503 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.995284 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:56.995476 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:56.995561 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:56.996707 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:56.996709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:56.996709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:56.996716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:56.997919 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.998302 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.998303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.998304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.998368 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:56.998566 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:56.998567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:56.998567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:56.998811 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:56.998940 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:56.999056 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 22:19:56.999143 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:19:57.000805 16902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:19:57.002338 16902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:19:57.008818 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.004725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.1.129.66:34519)
0923 22:19:57.004931 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:57.004936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:57.005215 (+ 279us) server_negotiation.cc:407] Connection header received
0923 22:19:57.005278 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:57.005280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:57.005296 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:57.005371 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:57.005586 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.006615 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.007760 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.008414 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.008497 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.008557 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:57.008699 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 22:19:57.008739 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260923 22:19:57.008822 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.004630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34519 (local address 127.0.0.1:47864)
0923 22:19:57.005153 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.005163 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:57.005209 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:57.005455 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:57.005456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:57.005462 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:57.005507 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.005509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.006751 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.006752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.007677 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.007679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.008496 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.008497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.008572 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.008588 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 22:19:57.008684 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 22:19:57.008739 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":43,"threads_started":1}
I20260923 22:19:57.008951 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.004403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44599 (local address 127.0.0.1:57394)
0923 22:19:57.004714 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.004757 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:57.004843 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:57.005049 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:57.005052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:57.005054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:57.005117 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.005118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.006479 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.006480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.007723 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.007725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.008688 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.008688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.008787 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.008790 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:19:57.008836 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 22:19:57.008880 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":22,"thread_start_us":64,"threads_started":1}
I20260923 22:19:57.008994 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.004747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.1.129.65:39141)
0923 22:19:57.004931 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:57.004936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:57.005065 (+ 129us) server_negotiation.cc:407] Connection header received
0923 22:19:57.005150 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:57.005152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:57.005171 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:57.005250 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:57.005473 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.006184 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.007736 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.008518 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.008568 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.008622 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:57.008874 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 22:19:57.008915 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260923 22:19:57.009194 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.004405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39141 (local address 127.0.0.1:37046)
0923 22:19:57.004976 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.004986 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:57.005074 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:57.005332 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:57.005333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:57.005335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:57.005387 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.005388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.006278 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.006279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.007638 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.007641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.008686 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.008687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.008775 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.008779 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:57.009085 (+ 306us) client_negotiation.cc:240] Negotiation successful
0923 22:19:57.009135 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260923 22:19:57.009215 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.004478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.1.129.67:44599)
0923 22:19:57.004662 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:57.004667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:57.004826 (+ 159us) server_negotiation.cc:407] Connection header received
0923 22:19:57.004930 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:57.004932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:57.004948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:57.005033 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:57.005173 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.006371 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.007808 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.008518 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.008604 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.008653 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:57.009101 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 22:19:57.009143 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260923 22:19:57.009893 17028 tablet_service.cc:1467] Processing CreateTablet for tablet ff4d7082bfd54edead063aa17c8d4b13 (DEFAULT_TABLE table=test-table [id=b40f5b838a2c4f98af2604242eb91f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:57.009893 17296 tablet_service.cc:1467] Processing CreateTablet for tablet ff4d7082bfd54edead063aa17c8d4b13 (DEFAULT_TABLE table=test-table [id=b40f5b838a2c4f98af2604242eb91f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:57.009938 17162 tablet_service.cc:1467] Processing CreateTablet for tablet ff4d7082bfd54edead063aa17c8d4b13 (DEFAULT_TABLE table=test-table [id=b40f5b838a2c4f98af2604242eb91f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:57.010226 17162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4d7082bfd54edead063aa17c8d4b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:57.010227 17296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4d7082bfd54edead063aa17c8d4b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:57.010227 17028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4d7082bfd54edead063aa17c8d4b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:57.012837 17376 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Bootstrap starting.
I20260923 22:19:57.013439 17376 tablet_bootstrap.cc:654] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:57.013430 17378 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: Bootstrap starting.
I20260923 22:19:57.014037 17376 log.cc:824] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:57.014101 17378 tablet_bootstrap.cc:654] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:57.014775 17377 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Bootstrap starting.
I20260923 22:19:57.014884 17376 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: No bootstrap required, opened a new log
I20260923 22:19:57.014950 17376 ts_tablet_manager.cc:1397] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:57.015391 17377 tablet_bootstrap.cc:654] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:57.015918 17377 log.cc:824] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:57.014586 17378 log.cc:824] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:57.016692 17377 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: No bootstrap required, opened a new log
I20260923 22:19:57.016763 17377 ts_tablet_manager.cc:1397] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:57.017153 17376 raft_consensus.cc:348] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.017330 17376 raft_consensus.cc:374] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:57.017364 17376 raft_consensus.cc:729] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22365fbe1a54772bbd55b848b9c12d1, State: Initialized, Role: FOLLOWER
I20260923 22:19:57.017481 17376 consensus_queue.cc:260] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.017746 17376 ts_tablet_manager.cc:1428] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:19:57.017777 17228 heartbeater.cc:503] Master 127.1.129.126:37263 was elected leader, sending a full tablet report...
I20260923 22:19:57.018769 17377 raft_consensus.cc:348] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.018997 17377 raft_consensus.cc:374] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:57.019052 17377 raft_consensus.cc:729] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f061a104f7194879bd12edbcfd22e8d9, State: Initialized, Role: FOLLOWER
I20260923 22:19:57.019112 17378 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: No bootstrap required, opened a new log
I20260923 22:19:57.019186 17378 ts_tablet_manager.cc:1397] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260923 22:19:57.019168 17377 consensus_queue.cc:260] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.019449 17377 ts_tablet_manager.cc:1428] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:19:57.019480 17094 heartbeater.cc:503] Master 127.1.129.126:37263 was elected leader, sending a full tablet report...
I20260923 22:19:57.020931 17378 raft_consensus.cc:348] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.021085 17378 raft_consensus.cc:374] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:57.021132 17378 raft_consensus.cc:729] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849e7c2d71d0463d98bdea2c2557e928, State: Initialized, Role: FOLLOWER
I20260923 22:19:57.021239 17378 consensus_queue.cc:260] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.021493 17378 ts_tablet_manager.cc:1428] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:57.021555 17362 heartbeater.cc:503] Master 127.1.129.126:37263 was elected leader, sending a full tablet report...
I20260923 22:19:57.059870 17382 raft_consensus.cc:482] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:57.060006 17382 raft_consensus.cc:504] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.060436 17382 leader_election.cc:288] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599), f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141)
I20260923 22:19:57.064404 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.060754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:35193 (local address 127.1.129.65:39141)
0923 22:19:57.060815 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:57.060817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:57.060827 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:19:57.060861 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:57.060862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:57.060871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:57.060917 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:57.061148 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.061898 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.063441 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.064014 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.064081 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.064089 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:57.064297 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 22:19:57.064333 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:19:57.064402 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.060774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:34971 (local address 127.1.129.67:44599)
0923 22:19:57.060838 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:57.060840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:57.060988 (+ 148us) server_negotiation.cc:407] Connection header received
0923 22:19:57.061031 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:57.061032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:57.061039 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:57.061081 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:57.061236 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.061925 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.063453 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.064004 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.064081 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.064089 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:57.064300 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 22:19:57.064337 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:19:57.064426 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.060624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39141 (local address 127.1.129.66:35193)
0923 22:19:57.060688 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.060701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:57.060762 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:57.060949 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:57.060950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:57.060953 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:57.061035 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.061037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.062001 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.062002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.063341 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.063345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.064103 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.064104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.064209 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.064211 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:19:57.064276 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 22:19:57.064337 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":74}
I20260923 22:19:57.064433 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.060624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44599 (local address 127.1.129.66:34971)
0923 22:19:57.060892 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.060919 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:57.060972 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:57.061103 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:57.061104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:57.061105 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:57.061171 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.061172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.062001 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.062002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.063360 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:57.063362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.064103 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.064104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:57.064204 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.064209 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:19:57.064276 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 22:19:57.064337 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":99,"thread_start_us":120,"threads_started":1}
I20260923 22:19:57.064848 17048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d7082bfd54edead063aa17c8d4b13" candidate_uuid: "e22365fbe1a54772bbd55b848b9c12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f061a104f7194879bd12edbcfd22e8d9" is_pre_election: true
I20260923 22:19:57.064870 17316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d7082bfd54edead063aa17c8d4b13" candidate_uuid: "e22365fbe1a54772bbd55b848b9c12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849e7c2d71d0463d98bdea2c2557e928" is_pre_election: true
I20260923 22:19:57.065008 17316 raft_consensus.cc:2454] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e22365fbe1a54772bbd55b848b9c12d1 in term 0.
I20260923 22:19:57.065006 17048 raft_consensus.cc:2454] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e22365fbe1a54772bbd55b848b9c12d1 in term 0.
I20260923 22:19:57.065250 17117 leader_election.cc:302] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849e7c2d71d0463d98bdea2c2557e928, e22365fbe1a54772bbd55b848b9c12d1; no voters:
I20260923 22:19:57.065374 17382 raft_consensus.cc:2781] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:57.065434 17382 raft_consensus.cc:482] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:57.065466 17382 raft_consensus.cc:3037] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:57.066334 17382 raft_consensus.cc:504] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.066491 17382 leader_election.cc:288] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [CANDIDATE]: Term 1 election: Requested vote from peers 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599), f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141)
I20260923 22:19:57.066679 17048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d7082bfd54edead063aa17c8d4b13" candidate_uuid: "e22365fbe1a54772bbd55b848b9c12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f061a104f7194879bd12edbcfd22e8d9"
I20260923 22:19:57.066672 17316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4d7082bfd54edead063aa17c8d4b13" candidate_uuid: "e22365fbe1a54772bbd55b848b9c12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849e7c2d71d0463d98bdea2c2557e928"
I20260923 22:19:57.066751 17316 raft_consensus.cc:3037] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:57.066751 17048 raft_consensus.cc:3037] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:57.067600 17048 raft_consensus.cc:2454] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e22365fbe1a54772bbd55b848b9c12d1 in term 1.
I20260923 22:19:57.067592 17316 raft_consensus.cc:2454] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e22365fbe1a54772bbd55b848b9c12d1 in term 1.
I20260923 22:19:57.067754 17117 leader_election.cc:302] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849e7c2d71d0463d98bdea2c2557e928, e22365fbe1a54772bbd55b848b9c12d1; no voters:
I20260923 22:19:57.067865 17382 raft_consensus.cc:2781] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:57.068028 17382 raft_consensus.cc:686] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 LEADER]: Becoming Leader. State: Replica: e22365fbe1a54772bbd55b848b9c12d1, State: Running, Role: LEADER
I20260923 22:19:57.068125 17382 consensus_queue.cc:237] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:19:57.068936 16902 catalog_manager.cc:5495] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66). New cstate: current_term: 1 leader_uuid: "e22365fbe1a54772bbd55b848b9c12d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } health_report { overall_health: HEALTHY } } }
W20260923 22:19:57.082978 17229 tablet.cc:2369] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:57.085625 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.083274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.1.129.66:34519)
0923 22:19:57.083315 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:57.083318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:57.083361 (+ 43us) server_negotiation.cc:407] Connection header received
0923 22:19:57.083463 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:57.083464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:57.083482 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:57.083549 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:57.083739 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.084281 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.085006 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:57.085179 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:57.085229 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:57.085351 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:57.085436 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:57.085489 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:57.085517 (+ 28us) server_negotiation.cc:299] Negotiation successful
0923 22:19:57.085531 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:19:57.087271 17048 raft_consensus.cc:1261] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Refusing update from remote peer e22365fbe1a54772bbd55b848b9c12d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:57.087278 17316 raft_consensus.cc:1261] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 FOLLOWER]: Refusing update from remote peer e22365fbe1a54772bbd55b848b9c12d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:57.087639 17386 consensus_queue.cc:1035] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:57.087759 17382 consensus_queue.cc:1035] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:57.092689 16903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:57.092795 16903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:57.093519 16903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Sending DeleteTablet for 3 replicas of tablet ff4d7082bfd54edead063aa17c8d4b13
I20260923 22:19:57.093866 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 16965
I20260923 22:19:57.093859 17162 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d7082bfd54edead063aa17c8d4b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:57 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:47864
I20260923 22:19:57.093860 17028 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d7082bfd54edead063aa17c8d4b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:57 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:37046
I20260923 22:19:57.094703 17296 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d7082bfd54edead063aa17c8d4b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:57 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:57394
I20260923 22:19:57.094739 17399 tablet_replica.cc:330] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: stopping tablet replica
I20260923 22:19:57.094853 17399 raft_consensus.cc:2229] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:19:57.094954 17399 raft_consensus.cc:2258] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:57.094954 17400 tablet_replica.cc:330] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: stopping tablet replica
I20260923 22:19:57.095089 17400 raft_consensus.cc:2229] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:57.095220 17400 raft_consensus.cc:2258] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:57.095559 17400 ts_tablet_manager.cc:1905] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:57.095943 17399 ts_tablet_manager.cc:1905] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:57.097322 17400 ts_tablet_manager.cc:1918] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:57.097400 17400 log.cc:1196] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/ff4d7082bfd54edead063aa17c8d4b13
I20260923 22:19:57.097683 17400 ts_tablet_manager.cc:1939] T ff4d7082bfd54edead063aa17c8d4b13 P 849e7c2d71d0463d98bdea2c2557e928: Deleting consensus metadata
I20260923 22:19:57.097803 17399 ts_tablet_manager.cc:1918] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:57.097916 17399 log.cc:1196] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/ff4d7082bfd54edead063aa17c8d4b13
I20260923 22:19:57.098062 16889 catalog_manager.cc:4841] TS 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599): tablet ff4d7082bfd54edead063aa17c8d4b13 (table test-table [id=b40f5b838a2c4f98af2604242eb91f05]) successfully deleted
I20260923 22:19:57.098327 17399 ts_tablet_manager.cc:1939] T ff4d7082bfd54edead063aa17c8d4b13 P e22365fbe1a54772bbd55b848b9c12d1: Deleting consensus metadata
I20260923 22:19:57.098732 16889 catalog_manager.cc:4841] TS e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519): tablet ff4d7082bfd54edead063aa17c8d4b13 (table test-table [id=b40f5b838a2c4f98af2604242eb91f05]) successfully deleted
W20260923 22:19:57.101250 16890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:39141 (error 108)
I20260923 22:19:57.101374 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 17099
I20260923 22:19:57.106462 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 17233
I20260923 22:19:57.106478 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.106352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39141 (local address 127.0.0.1:37058)
0923 22:19:57.106398 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.106437 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 22:19:57.106707 16890 catalog_manager.cc:4584] TS f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff4d7082bfd54edead063aa17c8d4b13: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39141: connect: Connection refused (error 111)
I20260923 22:19:57.110929 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 16871
I20260923 22:20:17.116683 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37263
--webserver_interface=127.1.129.126
--webserver_port=44583
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:17.200037 17402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:17.200268 17402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:17.200294 17402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:17.201516 17402 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 22:20:17.201560 17402 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 22:20:17.201575 17402 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 22:20:17.201589 17402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:17.201603 17402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:17.201618 17402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:17.201673 17402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:17.201694 17402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:17.203198 17402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:37263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:17.203436 17402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:17.203694 17402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:17.212399 17409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:17.212463 17411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:17.212437 17402 server_base.cc:1034] running on GCE node
W20260923 22:20:17.212405 17408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:17.213018 17402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:17.213342 17402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:17.214505 17402 hybrid_clock.cc:648] HybridClock initialized: now 1790202017214494 us; error 29 us; skew 500 ppm
Sep 23 22:20:17 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:17.217140 17402 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:20:17.217592 17402 webserver.cc:460] Webserver started at http://127.1.129.126:44583/ using document root <none> and password file <none>
I20260923 22:20:17.217859 17402 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:17.217945 17402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:17.219395 17402 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:20:17.220132 17418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.220337 17402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:17.220443 17402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "28c91cd2dcde4673bc3ca362449c693a"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "2469a915d0ab0c30da03d303ea3711eb"
server_key_iv: "ff40bf162f21508ee4cd1d859918e3ec"
server_key_version: "encryptionkey@0"
I20260923 22:20:17.220852 17402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:17.234941 17402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:17.236894 17402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:17.237056 17402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:17.241097 17402 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:37263
I20260923 22:20:17.241143 17470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:37263 every 8 connection(s)
I20260923 22:20:17.241564 17402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:17.242038 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 17402
I20260923 22:20:17.242316 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:39141
--local_ip_for_outbound_sockets=127.1.129.65
--tserver_master_addrs=127.1.129.126:37263
--webserver_port=40915
--webserver_interface=127.1.129.65
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:17.243387 17471 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:20:17.244096 17471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Bootstrap starting.
I20260923 22:20:17.252609 17471 log.cc:824] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:17.254834 17471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:20:17.255174 17471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Bootstrap complete.
I20260923 22:20:17.257481 17471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:20:17.257768 17471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c91cd2dcde4673bc3ca362449c693a, State: Initialized, Role: FOLLOWER
I20260923 22:20:17.257977 17471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:20:17.258064 17471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:17.258128 17471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:17.258183 17471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:20:17.259357 17471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:20:17.259492 17471 leader_election.cc:302] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c91cd2dcde4673bc3ca362449c693a; no voters:
I20260923 22:20:17.259717 17471 leader_election.cc:288] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 22:20:17.259764 17475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:20:17.259943 17475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 2 LEADER]: Becoming Leader. State: Replica: 28c91cd2dcde4673bc3ca362449c693a, State: Running, Role: LEADER
I20260923 22:20:17.260016 17471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:17.260097 17475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:20:17.260445 17476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "28c91cd2dcde4673bc3ca362449c693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } } }
I20260923 22:20:17.260557 17476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:17.260442 17475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c91cd2dcde4673bc3ca362449c693a. Latest consensus state: current_term: 2 leader_uuid: "28c91cd2dcde4673bc3ca362449c693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } } }
I20260923 22:20:17.260842 17475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:17.261088 17483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:17.261865 17483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:17.261969 17483 catalog_manager.cc:1259] Loaded cluster ID: 607fd89636f748058cb687d00841f282
I20260923 22:20:17.262051 17483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:20:17.263461 17483 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:17.263839 17483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Loaded TSK: 0
I20260923 22:20:17.276525 17483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Generated new TSK 1
I20260923 22:20:17.276652 17483 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:20:17.328966 17473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:17.329165 17473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:17.329190 17473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:17.330360 17473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:17.330402 17473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:17.330446 17473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:20:17.331948 17473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:39141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=40915
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:17.332190 17473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:17.332451 17473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:17.337368 17501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:17.337383 17498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:17.337632 17499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:17.338202 17473 server_base.cc:1034] running on GCE node
I20260923 22:20:17.338444 17473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:17.338754 17473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:17.339951 17473 hybrid_clock.cc:648] HybridClock initialized: now 1790202017339936 us; error 38 us; skew 500 ppm
Sep 23 22:20:17 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:17.342935 17473 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:20:17.343420 17473 webserver.cc:460] Webserver started at http://127.1.129.65:40915/ using document root <none> and password file <none>
I20260923 22:20:17.343662 17473 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:17.343742 17473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:17.345163 17473 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:17.345899 17508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.346076 17473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:17.346174 17473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "f061a104f7194879bd12edbcfd22e8d9"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "43bf9ae3e77c8e03699cd2d083765c27"
server_key_iv: "109ad55990af4ca1af635a6fb1ea5106"
server_key_version: "encryptionkey@0"
I20260923 22:20:17.346522 17473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:17.362246 17473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:17.364604 17473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:17.364811 17473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:17.365078 17473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:17.365548 17515 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:20:17.367529 17473 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:20:17.367592 17473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 22:20:17.367640 17473 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:20:17.368291 17473 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:20:17.368346 17473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:20:17.368425 17515 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Bootstrap starting.
I20260923 22:20:17.375475 17473 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39141
I20260923 22:20:17.375640 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39141 every 8 connection(s)
I20260923 22:20:17.375909 17473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:17.377283 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 17473
I20260923 22:20:17.377578 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:34519
--local_ip_for_outbound_sockets=127.1.129.66
--tserver_master_addrs=127.1.129.126:37263
--webserver_port=42413
--webserver_interface=127.1.129.66
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:17.379423 17515 log.cc:824] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:17.382072 17515 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:20:17.382372 17515 tablet_bootstrap.cc:492] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Bootstrap complete.
I20260923 22:20:17.383591 17515 ts_tablet_manager.cc:1397] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.013s
I20260923 22:20:17.384933 17515 raft_consensus.cc:348] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:20:17.385242 17515 raft_consensus.cc:729] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f061a104f7194879bd12edbcfd22e8d9, State: Initialized, Role: FOLLOWER
I20260923 22:20:17.385403 17515 consensus_queue.cc:260] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } } peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } }
I20260923 22:20:17.385660 17515 ts_tablet_manager.cc:1428] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 23 22:20:17 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:17.393486 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.376617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:49389)
0923 22:20:17.379758 (+ 3141us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:17.379811 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:17.380339 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:17.380907 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:17.380912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:17.381322 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:17.381605 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.381612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.383699 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.383701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.385157 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.385160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.386432 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.386434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.386586 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.387262 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:17.387273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:17.389252 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:17.391626 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:17.391633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:17.391635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:17.391928 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:17.392260 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:17.392261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:17.392261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:17.392375 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:17.392815 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:17.392821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:17.392973 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 22:20:17.393071 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 22:20:17.393281 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3090,"thread_start_us":90,"threads_started":1}
I20260923 22:20:17.393764 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.376684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:49389 (local address 127.1.129.126:37263)
0923 22:20:17.380533 (+ 3849us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.380541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.380570 (+ 29us) server_negotiation.cc:407] Connection header received
0923 22:20:17.380673 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.380677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.380719 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.380857 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:17.381746 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.383508 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.385414 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.386215 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.386330 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.389380 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:17.389384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:17.389385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:17.389397 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:17.391532 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.392060 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.392063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.392063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.392171 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.392439 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.392440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.392440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.392712 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:17.392788 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:17.393426 (+ 638us) server_negotiation.cc:299] Negotiation successful
0923 22:20:17.393564 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3802,"thread_start_us":77,"threads_started":1}
I20260923 22:20:17.394676 17623 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37263
I20260923 22:20:17.394765 17623 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:17.394932 17623 heartbeater.cc:511] Master 127.1.129.126:37263 requested a full tablet report, sending...
I20260923 22:20:17.395701 17435 ts_manager.cc:194] Registered new tserver with Master: f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141)
I20260923 22:20:17.396895 17435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:49389
I20260923 22:20:17.397814 17623 heartbeater.cc:503] Master 127.1.129.126:37263 was elected leader, sending a full tablet report...
Sep 23 22:20:17 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.65@KRBTEST.COM
I20260923 22:20:17.405323 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.396321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39141 (local address 127.0.0.1:41166)
0923 22:20:17.396726 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:17.396761 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:17.396824 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:17.397107 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:17.397278 (+ 171us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:17.397569 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:17.397621 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.397623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.398739 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.398741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.399579 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.399582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.400231 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.400232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.400304 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.400799 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:17.400803 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:17.402245 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:17.404261 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:17.404267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:17.404269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:17.404465 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:17.404678 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:17.404678 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:17.404679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:17.404739 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:17.405053 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:17.405059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:17.405100 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 22:20:17.405162 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 22:20:17.405254 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":82,"threads_started":1}
I20260923 22:20:17.405391 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.396553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.1.129.65:39141)
0923 22:20:17.396941 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.396946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.396954 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:17.396984 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.396986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.396999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.397051 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:17.397709 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.398604 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.399668 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.400142 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.400217 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.402332 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:17.402340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:17.402341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:17.402348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:17.404174 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.404546 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.404549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.404549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.404611 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.404808 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.404809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.404809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.404991 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:17.405069 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:17.405211 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 22:20:17.405303 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":80,"threads_started":1}
I20260923 22:20:17.405804 17557 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4d7082bfd54edead063aa17c8d4b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:57 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:41166
I20260923 22:20:17.406396 17632 tablet_replica.cc:330] stopping tablet replica
I20260923 22:20:17.406522 17632 raft_consensus.cc:2229] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:17.406601 17632 raft_consensus.cc:2258] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:17.406929 17632 ts_tablet_manager.cc:1905] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:17.409051 17632 ts_tablet_manager.cc:1918] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:17.409140 17632 log.cc:1196] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal/wals/ff4d7082bfd54edead063aa17c8d4b13
I20260923 22:20:17.409374 17632 ts_tablet_manager.cc:1939] T ff4d7082bfd54edead063aa17c8d4b13 P f061a104f7194879bd12edbcfd22e8d9: Deleting consensus metadata
I20260923 22:20:17.409875 17422 catalog_manager.cc:4841] TS f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141): tablet ff4d7082bfd54edead063aa17c8d4b13 (table test-table [id=b40f5b838a2c4f98af2604242eb91f05]) successfully deleted
W20260923 22:20:17.468662 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:17.468866 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:17.468889 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:17.469961 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:17.469995 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:17.470021 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:20:17.471501 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:34519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=42413
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:17.471688 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:17.471859 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:20:17.477049 17627 server_base.cc:1034] running on GCE node
W20260923 22:20:17.477011 17637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:17.477018 17638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:17.477008 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:17.477448 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:17.477705 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:17.478879 17627 hybrid_clock.cc:648] HybridClock initialized: now 1790202017478875 us; error 38 us; skew 500 ppm
Sep 23 22:20:17 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:17.481344 17627 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:20:17.481792 17627 webserver.cc:460] Webserver started at http://127.1.129.66:42413/ using document root <none> and password file <none>
I20260923 22:20:17.482018 17627 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:17.482105 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:17.483528 17627 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:17.484148 17647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.484311 17627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:20:17.484414 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "e22365fbe1a54772bbd55b848b9c12d1"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "f37397922359bf8f3ba70a156858285e"
server_key_iv: "cfcb012b27a20a9ef3fdd0467196bb08"
server_key_version: "encryptionkey@0"
I20260923 22:20:17.484754 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:17.506229 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:17.508487 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:17.508652 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:17.508908 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:17.509213 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:17.509266 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.509298 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:17.509330 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.515226 17627 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:34519
I20260923 22:20:17.515280 17760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:34519 every 8 connection(s)
I20260923 22:20:17.515609 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:20:17 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:17.524559 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 17627
I20260923 22:20:17.524885 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.516352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:46833 (local address 127.1.129.126:37263)
0923 22:20:17.516411 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.516413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.516631 (+ 218us) server_negotiation.cc:407] Connection header received
0923 22:20:17.517119 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.517120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.517131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.517185 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:17.518176 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.518752 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.519726 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.519947 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.520007 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.522067 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:17.522068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:17.522069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:17.522075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:17.523400 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.523896 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.523897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.523897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.523949 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.524159 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.524159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.524159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.524262 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:17.524311 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:17.524707 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 22:20:17.524801 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:20:17.524878 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:44599
--local_ip_for_outbound_sockets=127.1.129.67
--tserver_master_addrs=127.1.129.126:37263
--webserver_port=42265
--webserver_interface=127.1.129.67
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:17.525079 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.516358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:46833)
0923 22:20:17.516505 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:17.516540 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:17.517025 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:17.517239 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:17.517243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:17.517830 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:17.518032 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.518037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.518844 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.518845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.519638 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.519640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.520024 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.520025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.520124 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.520593 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:17.520601 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:17.521985 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:17.523513 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:17.523519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:17.523521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:17.523837 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:17.524008 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:17.524008 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:17.524009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:17.524107 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:17.524316 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:17.524321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:17.524501 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 22:20:17.524624 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 22:20:17.524810 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260923 22:20:17.525532 17761 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37263
I20260923 22:20:17.525630 17761 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:17.525815 17761 heartbeater.cc:511] Master 127.1.129.126:37263 requested a full tablet report, sending...
I20260923 22:20:17.526278 17435 ts_manager.cc:194] Registered new tserver with Master: e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519)
I20260923 22:20:17.526772 17435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:46833
W20260923 22:20:17.615283 17764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:17.615531 17764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:17.615573 17764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:17.616843 17764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:17.616915 17764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:17.616966 17764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:20:17.618566 17764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:44599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=42265
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:17.618888 17764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:17.619145 17764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:17.625310 17769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:17.625284 17772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:17.625288 17770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:17.625522 17764 server_base.cc:1034] running on GCE node
I20260923 22:20:17.625762 17764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:17.625962 17764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:17.627094 17764 hybrid_clock.cc:648] HybridClock initialized: now 1790202017627079 us; error 34 us; skew 500 ppm
Sep 23 22:20:17 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:17.629714 17764 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:20:17.630198 17764 webserver.cc:460] Webserver started at http://127.1.129.67:42265/ using document root <none> and password file <none>
I20260923 22:20:17.630394 17764 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:17.630461 17764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:17.631848 17764 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:17.632404 17779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.632557 17764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:17.632638 17764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "849e7c2d71d0463d98bdea2c2557e928"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "45321c8f1680527c266f95dccaf831c4"
server_key_iv: "18a505ba3a9ec543cbcaa981249b385c"
server_key_version: "encryptionkey@0"
I20260923 22:20:17.632969 17764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:17.651535 17764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:17.653623 17764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:17.653764 17764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:17.653968 17764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:17.654285 17764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:17.654320 17764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.654367 17764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:17.654394 17764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.660251 17764 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44599
I20260923 22:20:17.660344 17892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44599 every 8 connection(s)
I20260923 22:20:17.660678 17764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:20:17 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:17.669559 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.661511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:60625 (local address 127.1.129.126:37263)
0923 22:20:17.661599 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.661601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.661816 (+ 215us) server_negotiation.cc:407] Connection header received
0923 22:20:17.662290 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.662291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.662303 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.662378 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:17.663329 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.663807 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.664785 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.664974 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.665033 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.667223 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:17.667225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:17.667225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:17.667232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:17.668449 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.668809 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.668809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.668810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.668857 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.669056 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.669057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.669057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.669160 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:17.669208 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:17.669429 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 22:20:17.669492 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 22:20:17.669711 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.661497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:60625)
0923 22:20:17.661652 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:17.661692 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:17.662178 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:17.662432 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:17.662435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:17.663030 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:17.663222 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.663227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.663894 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.663895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.664687 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.664689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.665051 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.665052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.665148 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.665620 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:17.665628 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:17.667113 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:17.668518 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:17.668525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:17.668526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:17.668738 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:17.668914 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:17.668914 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:17.668915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:17.669005 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:17.669211 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:17.669215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:17.669324 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 22:20:17.669403 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 22:20:17.669518 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260923 22:20:17.670199 17893 heartbeater.cc:347] Connected to a master server at 127.1.129.126:37263
I20260923 22:20:17.670334 17893 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:17.670532 17893 heartbeater.cc:511] Master 127.1.129.126:37263 requested a full tablet report, sending...
I20260923 22:20:17.670593 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 17764
I20260923 22:20:17.670984 17435 ts_manager.cc:194] Registered new tserver with Master: 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599)
I20260923 22:20:17.671494 17435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:60625
I20260923 22:20:17.676999 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.671055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.1.129.126:37263)
0923 22:20:17.671092 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.671093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.671296 (+ 203us) server_negotiation.cc:407] Connection header received
0923 22:20:17.671350 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.671350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.671359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.671399 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:17.672089 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.672597 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.673306 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.673507 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.673594 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.674744 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:17.674746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:17.674746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:17.674752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:17.675873 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.676220 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.676221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.676222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.676279 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.676490 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.676491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.676491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.676602 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:17.676790 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:17.676825 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 22:20:17.676899 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:20:17.677682 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 22:20:17.680702 17366 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.129.126:37263)
I20260923 22:20:17.681010 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.678063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.1.129.126:37263)
0923 22:20:17.678107 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.678109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.678146 (+ 37us) server_negotiation.cc:407] Connection header received
0923 22:20:17.678226 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.678227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.678263 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.678357 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:17.678459 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.678987 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.679749 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.679952 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.680034 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.680210 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:17.680260 (+ 50us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 22:20:17.680540 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58210: token expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 22:20:17.681075 17626 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58210: token expired
I20260923 22:20:17.687956 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.680974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.1.129.126:37263)
0923 22:20:17.681146 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.681148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.681156 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:17.681175 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.681175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.681184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.681221 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:17.681967 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.682797 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.683852 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.684047 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.684127 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.685296 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:17.685298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:17.685298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:17.685305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:17.686827 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.687178 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.687179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.687179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.687231 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:17.687437 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:17.687437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:17.687438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:17.687572 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:17.687684 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:17.687784 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 22:20:17.687859 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260923 22:20:17.689148 17435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:20:17.690935 17435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:20:17.693397 17557 tablet_service.cc:1467] Processing CreateTablet for tablet 0026919a70ba4dc8bafbccec3b1265e9 (DEFAULT_TABLE table=test-table [id=a3942ebd20b04f2c97a93b4bfe079d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:17.693605 17557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026919a70ba4dc8bafbccec3b1265e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:17.695230 17515 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Bootstrap starting.
I20260923 22:20:17.695554 17515 tablet_bootstrap.cc:654] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:17.696363 17515 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: No bootstrap required, opened a new log
I20260923 22:20:17.696419 17515 ts_tablet_manager.cc:1397] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:17.696570 17515 raft_consensus.cc:348] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:17.696647 17515 raft_consensus.cc:374] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:17.696700 17515 raft_consensus.cc:729] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f061a104f7194879bd12edbcfd22e8d9, State: Initialized, Role: FOLLOWER
I20260923 22:20:17.696749 17515 consensus_queue.cc:260] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:17.696849 17515 ts_tablet_manager.cc:1428] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:17.697722 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.693577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.1.129.66:34519)
0923 22:20:17.693771 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.693776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.693787 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:20:17.693826 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.693828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.693844 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.693932 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:17.694086 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.694973 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.696553 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.697344 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.697408 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.697472 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:17.697608 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 22:20:17.697648 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260923 22:20:17.697711 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.693189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34519 (local address 127.0.0.1:44950)
0923 22:20:17.693237 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:17.693249 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:17.693285 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:17.693958 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:17.693959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:17.693961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:17.694008 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.694010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.695057 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.695059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.696301 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.696304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.697417 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.697418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.697491 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.697512 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 22:20:17.697591 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 22:20:17.697641 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 22:20:17.698364 17695 tablet_service.cc:1467] Processing CreateTablet for tablet 0026919a70ba4dc8bafbccec3b1265e9 (DEFAULT_TABLE table=test-table [id=a3942ebd20b04f2c97a93b4bfe079d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:17.698768 17695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026919a70ba4dc8bafbccec3b1265e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:17.698809 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.693322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44599 (local address 127.0.0.1:54712)
0923 22:20:17.693487 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:17.693497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:17.693544 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:17.695019 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:17.695020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:17.695022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:17.695065 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.695066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.696384 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.696385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.697747 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:17.697749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.698510 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.698511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:17.698576 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.698581 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:20:17.698674 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 22:20:17.698733 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260923 22:20:17.698846 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:17.693423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.1.129.67:44599)
0923 22:20:17.694771 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:17.694776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:17.694788 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:20:17.694825 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:17.694827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:17.694842 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:17.694925 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:17.695136 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.696294 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.697873 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:17.698410 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:17.698475 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:17.698512 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:17.698689 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 22:20:17.698735 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":102,"threads_started":1}
I20260923 22:20:17.701357 17903 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Bootstrap starting.
I20260923 22:20:17.702075 17903 tablet_bootstrap.cc:654] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:17.702040 17827 tablet_service.cc:1467] Processing CreateTablet for tablet 0026919a70ba4dc8bafbccec3b1265e9 (DEFAULT_TABLE table=test-table [id=a3942ebd20b04f2c97a93b4bfe079d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:17.702346 17827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026919a70ba4dc8bafbccec3b1265e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:17.702618 17903 log.cc:824] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:17.703431 17903 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: No bootstrap required, opened a new log
I20260923 22:20:17.703517 17903 ts_tablet_manager.cc:1397] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:20:17.705114 17903 raft_consensus.cc:348] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:17.705298 17903 raft_consensus.cc:374] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:17.705338 17903 raft_consensus.cc:729] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22365fbe1a54772bbd55b848b9c12d1, State: Initialized, Role: FOLLOWER
I20260923 22:20:17.705482 17905 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Bootstrap starting.
I20260923 22:20:17.705471 17903 consensus_queue.cc:260] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:17.705806 17903 ts_tablet_manager.cc:1428] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 22:20:17.705832 17761 heartbeater.cc:503] Master 127.1.129.126:37263 was elected leader, sending a full tablet report...
I20260923 22:20:17.706123 17905 tablet_bootstrap.cc:654] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:17.706727 17905 log.cc:824] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:17.707500 17905 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: No bootstrap required, opened a new log
I20260923 22:20:17.707558 17905 ts_tablet_manager.cc:1397] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:20:17.708974 17905 raft_consensus.cc:348] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:17.709126 17905 raft_consensus.cc:374] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:17.709151 17905 raft_consensus.cc:729] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849e7c2d71d0463d98bdea2c2557e928, State: Initialized, Role: FOLLOWER
I20260923 22:20:17.709293 17905 consensus_queue.cc:260] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:17.709546 17905 ts_tablet_manager.cc:1428] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:20:17.709579 17893 heartbeater.cc:503] Master 127.1.129.126:37263 was elected leader, sending a full tablet report...
W20260923 22:20:17.766163 17762 tablet.cc:2369] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:20:17.880173 17624 tablet.cc:2369] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:20:17.911404 17894 tablet.cc:2369] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:18.080785 17908 raft_consensus.cc:482] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:18.080912 17908 raft_consensus.cc:504] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:18.081279 17908 leader_election.cc:288] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519), f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141)
I20260923 22:20:18.081709 17906 raft_consensus.cc:482] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:18.081825 17906 raft_consensus.cc:504] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:18.082208 17906 leader_election.cc:288] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599), f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141)
I20260923 22:20:18.085078 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.081479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34519 (local address 127.1.129.67:37327)
0923 22:20:18.081551 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:18.081564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:18.081620 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:18.081866 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:18.081867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:18.081869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:18.081951 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:18.081953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.082916 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.082917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:18.084081 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:18.084083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.084825 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.084826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:18.084905 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.084910 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:20:18.084941 (+ 31us) client_negotiation.cc:240] Negotiation successful
0923 22:20:18.084990 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 22:20:18.085328 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.081562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39141 (local address 127.1.129.67:46013)
0923 22:20:18.081736 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:18.081747 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:18.081801 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:18.081996 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:18.081997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:18.081999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:18.082072 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:18.082073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.083003 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.083004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:18.084189 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:18.084191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.085075 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.085076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:18.085150 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.085155 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:20:18.085204 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:20:18.085244 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260923 22:20:18.085367 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.081553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:37327 (local address 127.1.129.66:34519)
0923 22:20:18.081603 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:18.081605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:18.081618 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:20:18.081764 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:18.081765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:18.081780 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:18.082014 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:18.082053 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.082825 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.084180 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.084737 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.085247 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.085257 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:18.085274 (+ 17us) server_negotiation.cc:299] Negotiation successful
0923 22:20:18.085308 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:20:18.085525 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.081721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:46013 (local address 127.1.129.65:39141)
0923 22:20:18.081829 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:18.081831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:18.081838 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:20:18.081870 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:18.081870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:18.081894 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:18.081975 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:18.082166 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.082916 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.084276 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.084896 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.084965 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.085041 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:18.085416 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 22:20:18.085458 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260923 22:20:18.085644 17715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026919a70ba4dc8bafbccec3b1265e9" candidate_uuid: "849e7c2d71d0463d98bdea2c2557e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22365fbe1a54772bbd55b848b9c12d1" is_pre_election: true
I20260923 22:20:18.085783 17715 raft_consensus.cc:2454] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849e7c2d71d0463d98bdea2c2557e928 in term 0.
I20260923 22:20:18.086001 17782 leader_election.cc:302] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849e7c2d71d0463d98bdea2c2557e928, e22365fbe1a54772bbd55b848b9c12d1; no voters:
I20260923 22:20:18.086133 17908 raft_consensus.cc:2781] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:18.086083 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026919a70ba4dc8bafbccec3b1265e9" candidate_uuid: "849e7c2d71d0463d98bdea2c2557e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f061a104f7194879bd12edbcfd22e8d9" is_pre_election: true
I20260923 22:20:18.086179 17908 raft_consensus.cc:482] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:18.086179 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.082585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:54413 (local address 127.1.129.65:39141)
0923 22:20:18.082711 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:18.082713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:18.082812 (+ 99us) server_negotiation.cc:407] Connection header received
0923 22:20:18.082899 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:18.082899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:18.082909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:18.082947 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:18.083154 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.083864 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.085210 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.085772 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.085819 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.085826 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:18.086044 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 22:20:18.086082 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260923 22:20:18.086174 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.082414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39141 (local address 127.1.129.66:54413)
0923 22:20:18.082759 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:18.082767 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:18.082834 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:18.083007 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:18.083008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:18.083009 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:18.083078 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:18.083080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.083960 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.083961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:18.085115 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:18.085118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.085885 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.085887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:18.085958 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.085961 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:18.086037 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 22:20:18.086079 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":33,"thread_start_us":58,"threads_started":1}
I20260923 22:20:18.086200 17908 raft_consensus.cc:3037] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:18.086254 17577 raft_consensus.cc:2454] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849e7c2d71d0463d98bdea2c2557e928 in term 0.
I20260923 22:20:18.086510 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026919a70ba4dc8bafbccec3b1265e9" candidate_uuid: "e22365fbe1a54772bbd55b848b9c12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f061a104f7194879bd12edbcfd22e8d9" is_pre_election: true
I20260923 22:20:18.086580 17577 raft_consensus.cc:2454] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e22365fbe1a54772bbd55b848b9c12d1 in term 0.
I20260923 22:20:18.086789 17651 leader_election.cc:302] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e22365fbe1a54772bbd55b848b9c12d1, f061a104f7194879bd12edbcfd22e8d9; no voters:
I20260923 22:20:18.086926 17906 raft_consensus.cc:2781] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:18.086983 17906 raft_consensus.cc:482] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:18.087018 17906 raft_consensus.cc:3037] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:18.087041 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.082454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:59037 (local address 127.1.129.67:44599)
0923 22:20:18.082551 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:18.082553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:18.082608 (+ 55us) server_negotiation.cc:407] Connection header received
0923 22:20:18.082721 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:18.082721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:18.082729 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:18.082755 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:18.082946 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.083678 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.085004 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.085529 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.085573 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.085580 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:18.086937 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0923 22:20:18.086974 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":78}
I20260923 22:20:18.087193 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.082333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44599 (local address 127.1.129.66:59037)
0923 22:20:18.082551 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:18.082560 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:18.082653 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:18.082806 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:18.082807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:18.082808 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:18.082872 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:18.082873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.083769 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.083770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:18.084916 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:18.084919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.086714 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.086715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:18.086792 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.086795 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:18.086929 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 22:20:18.086972 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":47,"threads_started":1}
I20260923 22:20:18.087292 17908 raft_consensus.cc:504] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:18.087497 17908 leader_election.cc:288] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [CANDIDATE]: Term 1 election: Requested vote from peers e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519), f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141)
I20260923 22:20:18.087595 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026919a70ba4dc8bafbccec3b1265e9" candidate_uuid: "849e7c2d71d0463d98bdea2c2557e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f061a104f7194879bd12edbcfd22e8d9"
I20260923 22:20:18.087669 17577 raft_consensus.cc:3037] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:18.087951 17906 raft_consensus.cc:504] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:18.088091 17906 leader_election.cc:288] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [CANDIDATE]: Term 1 election: Requested vote from peers 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599), f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141)
I20260923 22:20:18.088268 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026919a70ba4dc8bafbccec3b1265e9" candidate_uuid: "e22365fbe1a54772bbd55b848b9c12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f061a104f7194879bd12edbcfd22e8d9"
I20260923 22:20:18.088341 17715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026919a70ba4dc8bafbccec3b1265e9" candidate_uuid: "849e7c2d71d0463d98bdea2c2557e928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22365fbe1a54772bbd55b848b9c12d1"
I20260923 22:20:18.088420 17715 raft_consensus.cc:2379] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 849e7c2d71d0463d98bdea2c2557e928 in current term 1: Already voted for candidate e22365fbe1a54772bbd55b848b9c12d1 in this term.
I20260923 22:20:18.088551 17577 raft_consensus.cc:2454] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849e7c2d71d0463d98bdea2c2557e928 in term 1.
I20260923 22:20:18.088689 17847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026919a70ba4dc8bafbccec3b1265e9" candidate_uuid: "e22365fbe1a54772bbd55b848b9c12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849e7c2d71d0463d98bdea2c2557e928" is_pre_election: true
I20260923 22:20:18.088809 17783 leader_election.cc:302] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 849e7c2d71d0463d98bdea2c2557e928, f061a104f7194879bd12edbcfd22e8d9; no voters: e22365fbe1a54772bbd55b848b9c12d1
I20260923 22:20:18.088827 17847 raft_consensus.cc:2379] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e22365fbe1a54772bbd55b848b9c12d1 in current term 1: Already voted for candidate 849e7c2d71d0463d98bdea2c2557e928 in this term.
I20260923 22:20:18.088706 17846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026919a70ba4dc8bafbccec3b1265e9" candidate_uuid: "e22365fbe1a54772bbd55b848b9c12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849e7c2d71d0463d98bdea2c2557e928"
I20260923 22:20:18.088930 17908 raft_consensus.cc:2781] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:18.089010 17650 leader_election.cc:302] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e22365fbe1a54772bbd55b848b9c12d1; no voters: 849e7c2d71d0463d98bdea2c2557e928, f061a104f7194879bd12edbcfd22e8d9
I20260923 22:20:18.089098 17908 raft_consensus.cc:686] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 LEADER]: Becoming Leader. State: Replica: 849e7c2d71d0463d98bdea2c2557e928, State: Running, Role: LEADER
I20260923 22:20:18.089102 17906 raft_consensus.cc:2726] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260923 22:20:18.089192 17908 consensus_queue.cc:237] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:18.089944 17435 catalog_manager.cc:5495] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 reported cstate change: term changed from 0 to 1, leader changed from <none> to 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67). New cstate: current_term: 1 leader_uuid: "849e7c2d71d0463d98bdea2c2557e928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } health_report { overall_health: UNKNOWN } } }
I20260923 22:20:18.140702 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.138242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.1.129.67:44599)
0923 22:20:18.138308 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:18.138310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:18.138320 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:20:18.138364 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:18.138365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:18.138393 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:18.138473 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:18.138601 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.139128 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.139891 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:18.140088 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:18.140172 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:18.140355 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:18.140466 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:18.140568 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:18.140585 (+ 17us) server_negotiation.cc:299] Negotiation successful
0923 22:20:18.140605 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:20:18.142673 17577 raft_consensus.cc:1261] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Refusing update from remote peer 849e7c2d71d0463d98bdea2c2557e928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:18.142711 17715 raft_consensus.cc:1261] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Refusing update from remote peer 849e7c2d71d0463d98bdea2c2557e928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:18.142946 17908 consensus_queue.cc:1035] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:18.143092 17914 consensus_queue.cc:1035] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:18.146031 17918 mvcc.cc:205] Tried to move back new op lower bound from 7332667466309664768 to 7332667466093903872. Current Snapshot: MvccSnapshot[applied={T|T < 7332667466309664768 or (T in {7332667466309664768})}]
I20260923 22:20:18.146699 17920 mvcc.cc:205] Tried to move back new op lower bound from 7332667466309664768 to 7332667466093903872. Current Snapshot: MvccSnapshot[applied={T|T < 7332667466309664768 or (T in {7332667466309664768})}]
I20260923 22:20:18.147851 17435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:18.147950 17435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:18.148669 17435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Sending DeleteTablet for 3 replicas of tablet 0026919a70ba4dc8bafbccec3b1265e9
I20260923 22:20:18.149012 17557 tablet_service.cc:1514] Processing DeleteTablet for tablet 0026919a70ba4dc8bafbccec3b1265e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:18 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:41166
I20260923 22:20:18.149041 17827 tablet_service.cc:1514] Processing DeleteTablet for tablet 0026919a70ba4dc8bafbccec3b1265e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:18 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:54712
I20260923 22:20:18.149014 17695 tablet_service.cc:1514] Processing DeleteTablet for tablet 0026919a70ba4dc8bafbccec3b1265e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:18 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:44950
I20260923 22:20:18.149282 17927 tablet_replica.cc:330] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: stopping tablet replica
I20260923 22:20:18.149303 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 17473
I20260923 22:20:18.150769 17928 tablet_replica.cc:330] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: stopping tablet replica
I20260923 22:20:18.150885 17928 raft_consensus.cc:2229] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:18.150990 17928 raft_consensus.cc:2258] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:18.151144 17929 tablet_replica.cc:330] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: stopping tablet replica
I20260923 22:20:18.151242 17929 raft_consensus.cc:2229] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:20:18.151321 17929 raft_consensus.cc:2258] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:18.151897 17929 ts_tablet_manager.cc:1905] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:18.153714 17928 ts_tablet_manager.cc:1905] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:18.153797 17929 ts_tablet_manager.cc:1918] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:18.153949 17929 log.cc:1196] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/0026919a70ba4dc8bafbccec3b1265e9
I20260923 22:20:18.154296 17929 ts_tablet_manager.cc:1939] T 0026919a70ba4dc8bafbccec3b1265e9 P 849e7c2d71d0463d98bdea2c2557e928: Deleting consensus metadata
I20260923 22:20:18.154814 17421 catalog_manager.cc:4841] TS 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599): tablet 0026919a70ba4dc8bafbccec3b1265e9 (table test-table [id=a3942ebd20b04f2c97a93b4bfe079d60]) successfully deleted
W20260923 22:20:18.154953 17422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:39141 (error 108)
I20260923 22:20:18.155309 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 17627
I20260923 22:20:18.159926 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.159757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39141 (local address 127.0.0.1:41178)
0923 22:20:18.159823 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:18.159882 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 22:20:18.160075 17422 catalog_manager.cc:4584] TS f061a104f7194879bd12edbcfd22e8d9 (127.1.129.65:39141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0026919a70ba4dc8bafbccec3b1265e9: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39141: connect: Connection refused (error 111)
I20260923 22:20:18.161123 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 17764
I20260923 22:20:18.161193 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:18.161088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34519 (local address 127.0.0.1:44952)
0923 22:20:18.161130 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:18.161160 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.66:34519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 22:20:18.166256 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 17402
I20260923 22:20:38.171626 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37263
--webserver_interface=127.1.129.126
--webserver_port=44583
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:38.251295 17931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:38.251528 17931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:38.251552 17931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:38.252786 17931 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 22:20:38.252835 17931 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 22:20:38.252848 17931 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 22:20:38.252861 17931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:38.252873 17931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:38.252887 17931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:38.252898 17931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:38.252959 17931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:38.254483 17931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:37263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:38.254791 17931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:38.255057 17931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:38.259907 17939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:38.259955 17936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:38.259951 17937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:38.260114 17931 server_base.cc:1034] running on GCE node
I20260923 22:20:38.260430 17931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:38.260751 17931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:38.261925 17931 hybrid_clock.cc:648] HybridClock initialized: now 1790202038261923 us; error 54 us; skew 500 ppm
Sep 23 22:20:38 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:38.264506 17931 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:20:38.264983 17931 webserver.cc:460] Webserver started at http://127.1.129.126:44583/ using document root <none> and password file <none>
I20260923 22:20:38.265211 17931 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:38.265281 17931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:38.266741 17931 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:38.267464 17946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:38.267634 17931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:38.267740 17931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "28c91cd2dcde4673bc3ca362449c693a"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "2469a915d0ab0c30da03d303ea3711eb"
server_key_iv: "ff40bf162f21508ee4cd1d859918e3ec"
server_key_version: "encryptionkey@0"
I20260923 22:20:38.268116 17931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:38.295603 17931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:38.297504 17931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:38.297672 17931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:38.301772 17931 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:37263
I20260923 22:20:38.301811 17998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:37263 every 8 connection(s)
I20260923 22:20:38.302207 17931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:38.303755 17999 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:20:38.304358 17999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Bootstrap starting.
I20260923 22:20:38.306743 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 17931
I20260923 22:20:38.307045 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:39141
--local_ip_for_outbound_sockets=127.1.129.65
--tserver_master_addrs=127.1.129.126:37263
--webserver_port=40915
--webserver_interface=127.1.129.65
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:38.312817 17999 log.cc:824] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:38.315486 17999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:20:38.315840 17999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Bootstrap complete.
I20260923 22:20:38.318094 17999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:20:38.318341 17999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c91cd2dcde4673bc3ca362449c693a, State: Initialized, Role: FOLLOWER
I20260923 22:20:38.318519 17999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:20:38.318585 17999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:38.318665 17999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:38.318717 17999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 2 FOLLOWER]: Advancing to term 3
I20260923 22:20:38.319736 17999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:20:38.319861 17999 leader_election.cc:302] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c91cd2dcde4673bc3ca362449c693a; no voters:
I20260923 22:20:38.320073 17999 leader_election.cc:288] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 22:20:38.320108 18003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 3 FOLLOWER]: Leader election won for term 3
I20260923 22:20:38.320222 18003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [term 3 LEADER]: Becoming Leader. State: Replica: 28c91cd2dcde4673bc3ca362449c693a, State: Running, Role: LEADER
I20260923 22:20:38.320353 17999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:38.320385 18003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } }
I20260923 22:20:38.320672 18003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c91cd2dcde4673bc3ca362449c693a. Latest consensus state: current_term: 3 leader_uuid: "28c91cd2dcde4673bc3ca362449c693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } } }
I20260923 22:20:38.320678 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "28c91cd2dcde4673bc3ca362449c693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c91cd2dcde4673bc3ca362449c693a" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37263 } } }
I20260923 22:20:38.320885 18003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:38.320966 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:38.321458 18010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:38.322252 18010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:38.322347 18010 catalog_manager.cc:1259] Loaded cluster ID: 607fd89636f748058cb687d00841f282
I20260923 22:20:38.322371 18010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:20:38.323633 18010 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:38.323804 18010 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Loaded TSK: 0
I20260923 22:20:38.323845 18010 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28c91cd2dcde4673bc3ca362449c693a: Loaded TSK: 1
F20260923 22:20:38.324065 18010 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc5b21ad9ad google::LogMessage::Fail() at ??:0
@ 0x7fc5b21b088a google::LogMessage::SendToLog() at ??:0
@ 0x7fc5b21ad4de google::LogMessage::Flush() at ??:0
@ 0x7fc5b21aea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc5b491a104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc5b4917d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc5ba10b1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc5ba10b48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc5ba10b594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc5ba0d1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc5ba0dbad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc5b407b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc5b4070954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc5b4ed46db start_thread at ??:0
@ 0x7fc5b0cab61f clone at ??:0
W20260923 22:20:38.408576 18001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:38.408823 18001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:38.408862 18001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:38.410053 18001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:38.410126 18001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:38.410172 18001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:20:38.411767 18001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:39141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=40915
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:38.412048 18001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:38.412295 18001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:38.417366 18041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:38.417367 18040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:38.417558 18043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:38.417712 18001 server_base.cc:1034] running on GCE node
I20260923 22:20:38.418051 18001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:38.418341 18001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:38.419509 18001 hybrid_clock.cc:648] HybridClock initialized: now 1790202038419494 us; error 34 us; skew 500 ppm
Sep 23 22:20:38 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:38.422026 18001 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:20:38.422502 18001 webserver.cc:460] Webserver started at http://127.1.129.65:40915/ using document root <none> and password file <none>
I20260923 22:20:38.422778 18001 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:38.422848 18001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:38.424234 18001 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:38.424954 18050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:38.425148 18001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:38.425244 18001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "f061a104f7194879bd12edbcfd22e8d9"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "43bf9ae3e77c8e03699cd2d083765c27"
server_key_iv: "109ad55990af4ca1af635a6fb1ea5106"
server_key_version: "encryptionkey@0"
I20260923 22:20:38.425611 18001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:38.441987 18001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:38.444489 18001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:38.444659 18001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:38.444904 18001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:38.445356 18057 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:20:38.447418 18001 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:20:38.447494 18001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 22:20:38.447544 18001 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:20:38.448205 18001 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:20:38.448264 18001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:20:38.448334 18057 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Bootstrap starting.
I20260923 22:20:38.455565 18001 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39141
I20260923 22:20:38.455637 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39141 every 8 connection(s)
I20260923 22:20:38.456024 18001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:38.459487 18057 log.cc:824] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:38.462134 18057 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:20:38.462471 18057 tablet_bootstrap.cc:492] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Bootstrap complete.
I20260923 22:20:38.463724 18057 ts_tablet_manager.cc:1397] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260923 22:20:38.464205 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 18001
I20260923 22:20:38.464488 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:34519
--local_ip_for_outbound_sockets=127.1.129.66
--tserver_master_addrs=127.1.129.126:37263
--webserver_port=42413
--webserver_interface=127.1.129.66
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:38.465163 18057 raft_consensus.cc:348] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:38.465510 18057 raft_consensus.cc:729] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f061a104f7194879bd12edbcfd22e8d9, State: Initialized, Role: FOLLOWER
I20260923 22:20:38.465721 18057 consensus_queue.cc:260] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:38.466005 18057 ts_tablet_manager.cc:1428] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260923 22:20:38.530792 18167 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:20:38.456701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:59233)
0923 22:20:38.456870 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.456915 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:38.457397 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:38.530501 (+ 73104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
W20260923 22:20:38.531095 18165 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:37263 (0 consecutive failures): Network error: Failed to ping master at 127.1.129.126:37263: Client connection negotiation failed: client connection to 127.1.129.126:37263: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 22:20:38.531461 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:38.531337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:37497)
0923 22:20:38.531389 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.531435 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:20:38.531879 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:38.531709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:41061)
0923 22:20:38.531793 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.531853 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260923 22:20:38.532033 18165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:20:38.548071 18169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:38.548306 18169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:38.548331 18169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:38.549479 18169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:38.549516 18169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:38.549543 18169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:20:38.551417 18169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:34519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=42413
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:38.551644 18169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:38.551837 18169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:38.557120 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:38.557158 18169 server_base.cc:1034] running on GCE node
W20260923 22:20:38.557114 18176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:38.557176 18175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:38.557605 18169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:38.557865 18169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:38.559058 18169 hybrid_clock.cc:648] HybridClock initialized: now 1790202038559047 us; error 35 us; skew 500 ppm
Sep 23 22:20:38 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:38.565850 18169 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:20:38.566417 18169 webserver.cc:460] Webserver started at http://127.1.129.66:42413/ using document root <none> and password file <none>
I20260923 22:20:38.566730 18169 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:38.566803 18169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:38.568310 18169 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:38.569072 18185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:38.569242 18169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:38.569344 18169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "e22365fbe1a54772bbd55b848b9c12d1"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "f37397922359bf8f3ba70a156858285e"
server_key_iv: "cfcb012b27a20a9ef3fdd0467196bb08"
server_key_version: "encryptionkey@0"
I20260923 22:20:38.569725 18169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:38.591810 18169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:38.594306 18169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:38.594476 18169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:38.594748 18169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:38.595227 18192 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260923 22:20:38.596612 18192 ts_tablet_manager.cc:1852] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 22:20:38.596901 18192 ts_tablet_manager.cc:1905] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:38.599020 18192 ts_tablet_manager.cc:1918] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:38.599093 18192 log.cc:1196] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/0026919a70ba4dc8bafbccec3b1265e9
I20260923 22:20:38.599370 18192 ts_tablet_manager.cc:1939] T 0026919a70ba4dc8bafbccec3b1265e9 P e22365fbe1a54772bbd55b848b9c12d1: Deleting consensus metadata
I20260923 22:20:38.599596 18169 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 22:20:38.599678 18169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260923 22:20:38.599725 18169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:38.599756 18169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:38.605696 18169 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:34519
I20260923 22:20:38.605749 18299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:34519 every 8 connection(s)
I20260923 22:20:38.606052 18169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:20:38.607126 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:38.606769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:52735)
0923 22:20:38.606919 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.606990 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260923 22:20:38.607333 18300 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:37263 (0 consecutive failures): Network error: Failed to ping master at 127.1.129.126:37263: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
I20260923 22:20:38.607602 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:38.607517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:44757)
0923 22:20:38.607562 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.607587 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 22:20:38.607842 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:38.607782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:46581)
0923 22:20:38.607803 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.607826 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260923 22:20:38.607929 18300 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:20:38.610081 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 18169
I20260923 22:20:38.610368 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:44599
--local_ip_for_outbound_sockets=127.1.129.67
--tserver_master_addrs=127.1.129.126:37263
--webserver_port=42265
--webserver_interface=127.1.129.67
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:38.690231 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:38.690471 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:38.690495 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:38.691845 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:38.691890 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:38.691917 18303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:20:38.693399 18303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:44599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=42265
--tserver_master_addrs=127.1.129.126:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:38.693588 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:38.693758 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:38.698844 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:38.698779 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:38.698750 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:38.698769 18303 server_base.cc:1034] running on GCE node
I20260923 22:20:38.699184 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:38.699440 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:38.700590 18303 hybrid_clock.cc:648] HybridClock initialized: now 1790202038700570 us; error 33 us; skew 500 ppm
Sep 23 22:20:38 dist-test-slave-dmw7 krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:38.703286 18303 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:20:38.703847 18303 webserver.cc:460] Webserver started at http://127.1.129.67:42265/ using document root <none> and password file <none>
I20260923 22:20:38.704118 18303 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:38.704191 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:38.706032 18303 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:20:38.706743 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:38.706909 18303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:38.706998 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "849e7c2d71d0463d98bdea2c2557e928"
format_stamp: "Formatted at 2026-09-23 22:19:56 on dist-test-slave-dmw7"
server_key: "45321c8f1680527c266f95dccaf831c4"
server_key_iv: "18a505ba3a9ec543cbcaa981249b385c"
server_key_version: "encryptionkey@0"
I20260923 22:20:38.707325 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:38.727695 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:38.730095 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:38.730247 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:38.730506 18303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:38.730881 18303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:38.730939 18303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:38.730983 18303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:38.731009 18303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:38.736712 18303 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:44599
I20260923 22:20:38.736799 18431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:44599 every 8 connection(s)
I20260923 22:20:38.737156 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:20:38.738284 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:38.737818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:56019)
0923 22:20:38.737986 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.738101 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260923 22:20:38.738771 18432 heartbeater.cc:644] Failed to heartbeat to 127.1.129.126:37263 (0 consecutive failures): Network error: Failed to ping master at 127.1.129.126:37263: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
I20260923 22:20:38.739054 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:38.738962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:43349)
0923 22:20:38.739010 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.739037 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:20:38.739300 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:38.739244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:43337)
0923 22:20:38.739273 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:38.739290 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260923 22:20:38.739408 18432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:20:38.745420 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 18303
I20260923 22:20:39.532625 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:39.532380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:33025)
0923 22:20:39.532528 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:39.532564 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260923 22:20:39.608438 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:39.608257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:37755)
0923 22:20:39.608370 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:39.608405 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260923 22:20:39.739938 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:39.739703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:40367)
0923 22:20:39.739838 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:39.739874 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260923 22:20:39.816915 18439 raft_consensus.cc:482] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:39.817065 18439 raft_consensus.cc:504] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:39.817382 18439 leader_election.cc:288] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519), 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599)
Sep 23 22:20:39 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.66@KRBTEST.COM
Sep 23 22:20:39 dist-test-slave-dmw7 krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.67@KRBTEST.COM
I20260923 22:20:39.830765 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:39.817824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:53287 (local address 127.1.129.67:44599)
0923 22:20:39.817968 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:39.817976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:39.818005 (+ 29us) server_negotiation.cc:407] Connection header received
0923 22:20:39.818509 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:39.818512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:39.818573 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:39.818758 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:39.819792 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:39.821174 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:39.822716 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:39.823336 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:39.823453 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:39.826665 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:39.826670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:39.826671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:39.826684 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:39.828521 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:39.829181 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:39.829185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:39.829185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:39.829333 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:39.829675 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:39.829676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:39.829676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:39.830003 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:39.830092 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:39.830473 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 22:20:39.830650 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260923 22:20:39.830767 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:39.817703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:47237 (local address 127.1.129.66:34519)
0923 22:20:39.817846 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:39.817859 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:39.817922 (+ 63us) server_negotiation.cc:407] Connection header received
0923 22:20:39.818574 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:39.818578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:39.818619 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:39.818802 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:39.819792 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:39.821174 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:39.822716 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:39.823336 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:39.823453 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:39.826372 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:39.826376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:39.826377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:39.826389 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:39.828504 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:39.829181 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:39.829185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:39.829185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:39.829321 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:39.829675 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:39.829676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:39.829676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:39.830003 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:39.830092 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:39.830473 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 22:20:39.830626 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260923 22:20:39.830768 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:39.817579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34519 (local address 127.1.129.65:47237)
0923 22:20:39.817637 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:39.817649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:39.817708 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:39.818841 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:39.818845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:39.819389 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:39.819661 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:39.819669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:39.821284 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:39.821284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:39.822608 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:39.822611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:39.823472 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:39.823474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:39.823661 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:39.824321 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:39.824333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:39.826243 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:39.828596 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:39.828605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:39.828608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:39.829072 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:39.829406 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:39.829406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:39.829407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:39.829600 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:39.830086 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:39.830094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:39.830261 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 22:20:39.830402 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 22:20:39.830626 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":104}
I20260923 22:20:39.830822 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:39.817717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:44599 (local address 127.1.129.65:53287)
0923 22:20:39.817826 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:39.817835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:39.817877 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:39.818841 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:39.818845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:39.819389 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:39.819661 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:39.819669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:39.821281 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:39.821282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:39.822608 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:39.822611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:39.823473 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:39.823474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:39.823661 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:39.824322 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:39.824333 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:39.826527 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:39.828596 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:39.828605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:39.828608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:39.829072 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:39.829412 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:39.829413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:39.829413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:39.829600 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:39.830086 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:39.830094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:39.830261 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 22:20:39.830406 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 22:20:39.830705 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":247,"thread_start_us":54,"threads_started":1}
W20260923 22:20:39.831907 18053 leader_election.cc:341] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519): Not found: Tablet not found: 0026919a70ba4dc8bafbccec3b1265e9
W20260923 22:20:39.832199 18053 leader_election.cc:341] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599): Not found: Tablet not found: 0026919a70ba4dc8bafbccec3b1265e9
I20260923 22:20:39.832255 18053 leader_election.cc:302] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f061a104f7194879bd12edbcfd22e8d9; no voters: 849e7c2d71d0463d98bdea2c2557e928, e22365fbe1a54772bbd55b848b9c12d1
I20260923 22:20:39.832360 18439 raft_consensus.cc:2726] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:20:40.533315 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:40.533103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:40945)
0923 22:20:40.533230 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:40.533269 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260923 22:20:40.609163 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:40.608876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:45745)
0923 22:20:40.609030 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:40.609071 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260923 22:20:40.740576 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:40.740366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:35419)
0923 22:20:40.740506 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:40.740541 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260923 22:20:41.534011 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:41.533776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:37243)
0923 22:20:41.533916 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:41.533963 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260923 22:20:41.567152 18447 raft_consensus.cc:482] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:41.567255 18447 raft_consensus.cc:504] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:41.567451 18447 leader_election.cc:288] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519), 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599)
W20260923 22:20:41.567835 18053 leader_election.cc:341] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519): Not found: Tablet not found: 0026919a70ba4dc8bafbccec3b1265e9
W20260923 22:20:41.567917 18053 leader_election.cc:341] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599): Not found: Tablet not found: 0026919a70ba4dc8bafbccec3b1265e9
I20260923 22:20:41.567934 18053 leader_election.cc:302] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f061a104f7194879bd12edbcfd22e8d9; no voters: 849e7c2d71d0463d98bdea2c2557e928, e22365fbe1a54772bbd55b848b9c12d1
I20260923 22:20:41.568014 18447 raft_consensus.cc:2726] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:20:41.609876 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:41.609646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:57769)
0923 22:20:41.609801 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:41.609829 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260923 22:20:41.741199 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:41.741015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:44379)
0923 22:20:41.741126 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:41.741161 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260923 22:20:42.534775 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.534526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:41181)
0923 22:20:42.534699 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.534732 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":59,"threads_started":1}
I20260923 22:20:42.610757 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.610439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:55855)
0923 22:20:42.610616 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.610717 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260923 22:20:42.742026 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.741715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:55721)
0923 22:20:42.741915 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.741967 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":59,"threads_started":1}
I20260923 22:20:43.535481 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.535252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:47587)
0923 22:20:43.535392 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:43.535435 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260923 22:20:43.611438 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.611250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:52221)
0923 22:20:43.611362 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:43.611396 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260923 22:20:43.742832 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.742564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:60621)
0923 22:20:43.742746 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:43.742788 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
I20260923 22:20:43.943106 18456 raft_consensus.cc:482] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:43.943202 18456 raft_consensus.cc:504] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:43.943356 18456 leader_election.cc:288] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519), 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599)
W20260923 22:20:43.943753 18053 leader_election.cc:341] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519): Not found: Tablet not found: 0026919a70ba4dc8bafbccec3b1265e9
W20260923 22:20:43.943822 18053 leader_election.cc:341] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599): Not found: Tablet not found: 0026919a70ba4dc8bafbccec3b1265e9
I20260923 22:20:43.943840 18053 leader_election.cc:302] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f061a104f7194879bd12edbcfd22e8d9; no voters: 849e7c2d71d0463d98bdea2c2557e928, e22365fbe1a54772bbd55b848b9c12d1
I20260923 22:20:43.943934 18456 raft_consensus.cc:2726] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:20:44.536247 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:44.536006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:45233)
0923 22:20:44.536156 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:44.536200 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260923 22:20:44.612093 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:44.611853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:55669)
0923 22:20:44.611986 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:44.612046 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260923 22:20:44.743598 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:44.743367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:49581)
0923 22:20:44.743511 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:44.743553 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260923 22:20:45.536980 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:45.536703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:55835)
0923 22:20:45.536857 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:45.536920 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260923 22:20:45.612787 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:45.612583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:41247)
0923 22:20:45.612704 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:45.612744 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260923 22:20:45.744318 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:45.744104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:59037)
0923 22:20:45.744231 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:45.744273 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260923 22:20:46.537820 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:46.537508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:43423)
0923 22:20:46.537732 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:46.537774 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260923 22:20:46.613539 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:46.613354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:40887)
0923 22:20:46.613468 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:46.613501 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260923 22:20:46.745048 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:46.744834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:48131)
0923 22:20:46.744966 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:46.745002 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260923 22:20:47.538539 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:47.538338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:48455)
0923 22:20:47.538456 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:47.538496 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260923 22:20:47.614203 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:47.613989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:34577)
0923 22:20:47.614123 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:47.614160 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260923 22:20:47.745747 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:47.745510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:42647)
0923 22:20:47.745654 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:47.745697 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
I20260923 22:20:48.539324 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:48.539079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:51023)
0923 22:20:48.539224 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:48.539255 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260923 22:20:48.548141 18470 raft_consensus.cc:482] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:48.548256 18470 raft_consensus.cc:504] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22365fbe1a54772bbd55b848b9c12d1" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34519 } } peers { permanent_uuid: "849e7c2d71d0463d98bdea2c2557e928" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 44599 } } peers { permanent_uuid: "f061a104f7194879bd12edbcfd22e8d9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39141 } }
I20260923 22:20:48.548473 18470 leader_election.cc:288] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519), 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599)
W20260923 22:20:48.548919 18053 leader_election.cc:341] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 849e7c2d71d0463d98bdea2c2557e928 (127.1.129.67:44599): Not found: Tablet not found: 0026919a70ba4dc8bafbccec3b1265e9
W20260923 22:20:48.548985 18053 leader_election.cc:341] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e22365fbe1a54772bbd55b848b9c12d1 (127.1.129.66:34519): Not found: Tablet not found: 0026919a70ba4dc8bafbccec3b1265e9
I20260923 22:20:48.549000 18053 leader_election.cc:302] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f061a104f7194879bd12edbcfd22e8d9; no voters: 849e7c2d71d0463d98bdea2c2557e928, e22365fbe1a54772bbd55b848b9c12d1
I20260923 22:20:48.549078 18470 raft_consensus.cc:2726] T 0026919a70ba4dc8bafbccec3b1265e9 P f061a104f7194879bd12edbcfd22e8d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:20:48.614931 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:48.614734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:44691)
0923 22:20:48.614851 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:48.614886 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260923 22:20:48.746567 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:48.746246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:56855)
0923 22:20:48.746397 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:48.746437 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260923 22:20:49.540024 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:49.539793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:59773)
0923 22:20:49.539938 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:49.539977 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260923 22:20:49.615658 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:49.615414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:36921)
0923 22:20:49.615569 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:49.615610 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
I20260923 22:20:49.747210 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:49.746992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:57153)
0923 22:20:49.747132 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:49.747168 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260923 22:20:50.540669 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:50.540454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:38731)
0923 22:20:50.540586 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:50.540624 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260923 22:20:50.616420 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:50.616176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:40145)
0923 22:20:50.616330 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:50.616368 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260923 22:20:50.747922 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:50.747672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:44645)
0923 22:20:50.747816 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:50.747863 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260923 22:20:51.541468 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:51.541234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:39959)
0923 22:20:51.541373 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:51.541421 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260923 22:20:51.617053 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:51.616842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:51717)
0923 22:20:51.616977 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:51.617009 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260923 22:20:51.748654 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:51.748418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:59807)
0923 22:20:51.748565 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:51.748608 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260923 22:20:52.542250 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:52.542003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:43163)
0923 22:20:52.542163 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:52.542205 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260923 22:20:52.617727 18483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:52.617503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:36439)
0923 22:20:52.617646 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:52.617680 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260923 22:20:52.749418 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:52.749190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:47471)
0923 22:20:52.749324 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:52.749366 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260923 22:20:53.543027 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:53.542843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.65:44327)
0923 22:20:53.542955 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:53.542984 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260923 22:20:53.618753 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:53.618481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.66:40405)
0923 22:20:53.618621 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:53.618716 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":58,"threads_started":1}
I20260923 22:20:53.750226 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:53.750002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:37263 (local address 127.1.129.67:60499)
0923 22:20:53.750142 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:53.750178 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.126:37263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260923 22:20:53.750600 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 18001
I20260923 22:20:53.755476 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 18169
I20260923 22:20:53.760725 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 18303
2026-09-23T22:20:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58563 ms)
[----------] 30 tests from SecurityITest (234557 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:20:53 dist-test-slave-dmw7 krb5kdc[18490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:20:53 dist-test-slave-dmw7 krb5kdc[18490](info): set up 2 sockets
Sep 23 22:20:53 dist-test-slave-dmw7 krb5kdc[18490](info): commencing operation
krb5kdc: starting...
W20260923 22:20:54.794291 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:20:54 dist-test-slave-dmw7 krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:20:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:20:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:20:54.907073 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:40533
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:40533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:54.985641 18503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:54.985874 18503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:54.985911 18503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:54.987051 18503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:54.987119 18503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:54.987142 18503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:54.987164 18503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:54.987193 18503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:54.988643 18503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:40533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:54.988914 18503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:54.989133 18503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:54.994055 18511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:54.994180 18509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:54.994076 18508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:54.994118 18503 server_base.cc:1034] running on GCE node
I20260923 22:20:54.994725 18503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:54.995002 18503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:54.996168 18503 hybrid_clock.cc:648] HybridClock initialized: now 1790202054996162 us; error 35 us; skew 500 ppm
Sep 23 22:20:54 dist-test-slave-dmw7 krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202054, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:54.998567 18503 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:20:54.999019 18503 webserver.cc:460] Webserver started at http://127.1.129.126:42235/ using document root <none> and password file <none>
I20260923 22:20:54.999241 18503 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:54.999308 18503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:54.999462 18503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:55.000517 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "5ef4614feba54d689b3a7bcda2f4d7d0"
format_stamp: "Formatted at 2026-09-23 22:20:54 on dist-test-slave-dmw7"
server_key: "51715638e46a743c8014c24c0a03a118"
server_key_iv: "a9d190e7ed799278bf8087fb451dea9f"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.000880 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "5ef4614feba54d689b3a7bcda2f4d7d0"
format_stamp: "Formatted at 2026-09-23 22:20:54 on dist-test-slave-dmw7"
server_key: "51715638e46a743c8014c24c0a03a118"
server_key_iv: "a9d190e7ed799278bf8087fb451dea9f"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.002225 18503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:20:55.003167 18518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.003386 18503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:20:55.003500 18503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "5ef4614feba54d689b3a7bcda2f4d7d0"
format_stamp: "Formatted at 2026-09-23 22:20:54 on dist-test-slave-dmw7"
server_key: "51715638e46a743c8014c24c0a03a118"
server_key_iv: "a9d190e7ed799278bf8087fb451dea9f"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.003587 18503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:55.016067 18503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:55.017904 18503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:55.018075 18503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:55.022183 18503 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:40533
I20260923 22:20:55.022238 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:40533 every 8 connection(s)
I20260923 22:20:55.022591 18503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:55.023488 18571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:55.025835 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0: Bootstrap starting.
I20260923 22:20:55.026461 18571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:55.026814 18571 log.cc:824] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:55.027494 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0: No bootstrap required, opened a new log
I20260923 22:20:55.029095 18571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef4614feba54d689b3a7bcda2f4d7d0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40533 } }
I20260923 22:20:55.029234 18571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:55.029260 18571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef4614feba54d689b3a7bcda2f4d7d0, State: Initialized, Role: FOLLOWER
I20260923 22:20:55.029441 18571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef4614feba54d689b3a7bcda2f4d7d0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40533 } }
I20260923 22:20:55.029508 18571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:55.029531 18571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:55.029608 18571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:55.030148 18571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef4614feba54d689b3a7bcda2f4d7d0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40533 } }
I20260923 22:20:55.030263 18571 leader_election.cc:302] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ef4614feba54d689b3a7bcda2f4d7d0; no voters:
I20260923 22:20:55.030462 18571 leader_election.cc:288] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:20:55.030508 18574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:55.030694 18574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [term 1 LEADER]: Becoming Leader. State: Replica: 5ef4614feba54d689b3a7bcda2f4d7d0, State: Running, Role: LEADER
I20260923 22:20:55.030828 18574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef4614feba54d689b3a7bcda2f4d7d0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40533 } }
I20260923 22:20:55.031204 18575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ef4614feba54d689b3a7bcda2f4d7d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef4614feba54d689b3a7bcda2f4d7d0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40533 } } }
I20260923 22:20:55.031353 18576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ef4614feba54d689b3a7bcda2f4d7d0. Latest consensus state: current_term: 1 leader_uuid: "5ef4614feba54d689b3a7bcda2f4d7d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef4614feba54d689b3a7bcda2f4d7d0" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40533 } } }
I20260923 22:20:55.031399 18575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:55.031433 18576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:55.031466 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 18503
I20260923 22:20:55.031575 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:20:55.031783 18578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:55.031822 1541 external_mini_cluster.cc:1428] Setting key 7b5b7c12ce405e16aa3ee86620298b32
I20260923 22:20:55.031221 18571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:55.032284 18578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:55.035064 18578 catalog_manager.cc:1347] Generated new cluster ID: 0531a5e21d824854851d568092d6413a
I20260923 22:20:55.035117 18578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:20:55 dist-test-slave-dmw7 krb5kdc[18490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:55.043663 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.032301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.1.129.126:40533)
0923 22:20:55.032484 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.032493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.032531 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:20:55.032666 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.032674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.032754 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.032919 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:55.034062 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.035478 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.036530 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.037154 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.037271 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.040131 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:55.040135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:55.040136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:55.040147 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:55.042015 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.042426 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.042428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.042429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.042540 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.042779 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.042780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.042780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.043070 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:55.043223 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.043323 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.043448 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260923 22:20:55.046368 18578 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:20:55.047070 18578 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:55.051621 18578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0: Generated new TSK 0
I20260923 22:20:55.051774 18578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:20:55.077133 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40533
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:55.162424 18597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:55.162631 18597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:55.162720 18597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:55.163913 18597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:55.163965 18597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:55.163993 18597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:20:55.165417 18597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:55.165658 18597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:55.165906 18597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:55.170903 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:55.170924 18603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:55.170934 18602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:55.171082 18597 server_base.cc:1034] running on GCE node
I20260923 22:20:55.171329 18597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:55.171628 18597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:55.172781 18597 hybrid_clock.cc:648] HybridClock initialized: now 1790202055172776 us; error 38 us; skew 500 ppm
Sep 23 22:20:55 dist-test-slave-dmw7 krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:55.175468 18597 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:20:55.175918 18597 webserver.cc:460] Webserver started at http://127.1.129.65:36285/ using document root <none> and password file <none>
I20260923 22:20:55.176163 18597 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:55.176235 18597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:55.176380 18597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:55.177467 18597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "af0d5dd0275242399bdb29e6241bce90"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "4c21e41d93d27e8d661a71766cbe6466"
server_key_iv: "523f1d940fb32dd87473dcd1e2bdc0e6"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.177836 18597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "af0d5dd0275242399bdb29e6241bce90"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "4c21e41d93d27e8d661a71766cbe6466"
server_key_iv: "523f1d940fb32dd87473dcd1e2bdc0e6"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.179245 18597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:20:55.180047 18612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.180265 18597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:55.180371 18597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "af0d5dd0275242399bdb29e6241bce90"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "4c21e41d93d27e8d661a71766cbe6466"
server_key_iv: "523f1d940fb32dd87473dcd1e2bdc0e6"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.180446 18597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:55.203243 18597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:55.205241 18597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:55.205412 18597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:55.205660 18597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:55.205989 18597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:55.206051 18597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.206111 18597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:55.206142 18597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.212071 18597 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:39725
I20260923 22:20:55.212118 18725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:39725 every 8 connection(s)
I20260923 22:20:55.212473 18597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:20:55 dist-test-slave-dmw7 krb5kdc[18490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:55.221822 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.213300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:40141 (local address 127.1.129.126:40533)
0923 22:20:55.213405 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.213407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.213580 (+ 173us) server_negotiation.cc:407] Connection header received
0923 22:20:55.214041 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.214043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.214053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.214113 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:55.215178 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.215701 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.216731 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.216977 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.217049 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.219418 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:55.219419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:55.219420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:55.219426 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:55.220623 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.221000 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.221001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.221001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.221058 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.221282 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.221283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.221284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.221390 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:55.221456 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.221687 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.221759 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 22:20:55.222000 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.213248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:40533 (local address 127.1.129.65:40141)
0923 22:20:55.213418 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.213460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:55.213953 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:55.214176 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:55.214179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:55.214861 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:55.215070 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.215076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.215793 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.215794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.216629 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.216630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.217091 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.217093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.217226 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.217826 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:55.217835 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:55.219324 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:55.220694 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:55.220700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:55.220702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:55.220913 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:55.221128 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:55.221128 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:55.221128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:55.221225 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:55.221458 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:55.221462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:55.221571 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 22:20:55.221657 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 22:20:55.221776 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260923 22:20:55.222314 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 18597
I20260923 22:20:55.222440 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:20:55.222546 18726 heartbeater.cc:347] Connected to a master server at 127.1.129.126:40533
I20260923 22:20:55.222673 18726 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:55.222721 1541 external_mini_cluster.cc:1428] Setting key 660bce37b9f854a74c305b5c46944e4c
I20260923 22:20:55.222877 18726 heartbeater.cc:511] Master 127.1.129.126:40533 requested a full tablet report, sending...
I20260923 22:20:55.223420 18535 ts_manager.cc:194] Registered new tserver with Master: af0d5dd0275242399bdb29e6241bce90 (127.1.129.65:39725)
I20260923 22:20:55.224025 18535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:40141
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:20:55.247861 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40533
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:55.327486 18731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:55.327680 18731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:55.327702 18731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:55.328840 18731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:55.328881 18731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:55.328907 18731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:20:55.330328 18731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:55.330571 18731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:55.330832 18731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:55.335920 18739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:55.335942 18736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:55.336126 18731 server_base.cc:1034] running on GCE node
W20260923 22:20:55.335942 18737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:55.336468 18731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:55.336721 18731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:55.337920 18731 hybrid_clock.cc:648] HybridClock initialized: now 1790202055337884 us; error 49 us; skew 500 ppm
Sep 23 22:20:55 dist-test-slave-dmw7 krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:55.340513 18731 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:20:55.340955 18731 webserver.cc:460] Webserver started at http://127.1.129.66:37709/ using document root <none> and password file <none>
I20260923 22:20:55.341199 18731 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:55.341271 18731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:55.341434 18731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:55.342512 18731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "31c84d2026554ba8b97a56ae9b944d4b"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "e5067e152b1da6f8bcd57de55f0d000b"
server_key_iv: "b015997212c8a7dffe4e6a85821b2957"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.342936 18731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "31c84d2026554ba8b97a56ae9b944d4b"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "e5067e152b1da6f8bcd57de55f0d000b"
server_key_iv: "b015997212c8a7dffe4e6a85821b2957"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.344290 18731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:55.345137 18746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.345350 18731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:55.345458 18731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "31c84d2026554ba8b97a56ae9b944d4b"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "e5067e152b1da6f8bcd57de55f0d000b"
server_key_iv: "b015997212c8a7dffe4e6a85821b2957"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.345549 18731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:55.367465 18731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:55.369323 18731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:55.369490 18731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:55.369721 18731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:55.370031 18731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:55.370086 18731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.370127 18731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:55.370155 18731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.375911 18731 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:45869
I20260923 22:20:55.375970 18859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:45869 every 8 connection(s)
I20260923 22:20:55.376303 18731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:20:55 dist-test-slave-dmw7 krb5kdc[18490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:55.382560 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 18731
I20260923 22:20:55.382691 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:20:55.382927 1541 external_mini_cluster.cc:1428] Setting key cf2c543f01378cd296ff57cf75272a21
I20260923 22:20:55.386001 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.377047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:37333 (local address 127.1.129.126:40533)
0923 22:20:55.377120 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.377122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.377335 (+ 213us) server_negotiation.cc:407] Connection header received
0923 22:20:55.377851 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.377853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.377862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.377924 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:55.378901 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.379401 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.380333 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.380510 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.380557 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.382898 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:55.382900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:55.382900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:55.382908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:55.384519 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.385003 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.385004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.385005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.385073 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.385330 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.385331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.385331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.385475 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:55.385545 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.385801 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.385889 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:20:55.386124 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.377034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:40533 (local address 127.1.129.66:37333)
0923 22:20:55.377189 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.377248 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:55.377766 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:55.377979 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:55.377982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:55.378559 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:55.378828 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.378833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.379487 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.379488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.380254 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.380255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.380578 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.380579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.380691 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.381148 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:55.381155 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:55.382676 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:55.384592 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:55.384599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:55.384601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:55.384923 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:55.385152 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:55.385153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:55.385153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:55.385266 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:55.385548 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:55.385553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:55.385668 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 22:20:55.385759 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 22:20:55.385876 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260923 22:20:55.386588 18860 heartbeater.cc:347] Connected to a master server at 127.1.129.126:40533
I20260923 22:20:55.386746 18860 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:55.386922 18860 heartbeater.cc:511] Master 127.1.129.126:40533 requested a full tablet report, sending...
I20260923 22:20:55.387295 18535 ts_manager.cc:194] Registered new tserver with Master: 31c84d2026554ba8b97a56ae9b944d4b (127.1.129.66:45869)
I20260923 22:20:55.387702 18535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:37333
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:20:55.407351 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40533
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:55.487900 18865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:55.488140 18865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:55.488179 18865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:55.489315 18865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:55.489382 18865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:55.489427 18865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:20:55.490904 18865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:55.491170 18865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:55.491401 18865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:55.496516 18870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:55.496546 18865 server_base.cc:1034] running on GCE node
W20260923 22:20:55.496539 18871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:55.496516 18873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:55.496897 18865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:55.497150 18865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:55.498291 18865 hybrid_clock.cc:648] HybridClock initialized: now 1790202055498271 us; error 36 us; skew 500 ppm
Sep 23 22:20:55 dist-test-slave-dmw7 krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:55.500633 18865 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:20:55.501094 18865 webserver.cc:460] Webserver started at http://127.1.129.67:35507/ using document root <none> and password file <none>
I20260923 22:20:55.501323 18865 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:55.501366 18865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:55.501519 18865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:55.502482 18865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "9c68f99f4c134f68bb82d354cc5f983d"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "c03f6860d59eb821ff9a7befc98342ed"
server_key_iv: "3e77d1b72ef44d8707b453b4b8fcb6d2"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.502835 18865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "9c68f99f4c134f68bb82d354cc5f983d"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "c03f6860d59eb821ff9a7befc98342ed"
server_key_iv: "3e77d1b72ef44d8707b453b4b8fcb6d2"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.504068 18865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:55.504724 18880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.504910 18865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:20:55.505007 18865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "9c68f99f4c134f68bb82d354cc5f983d"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "c03f6860d59eb821ff9a7befc98342ed"
server_key_iv: "3e77d1b72ef44d8707b453b4b8fcb6d2"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.505080 18865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:55.517287 18865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:55.519434 18865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:55.519577 18865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:55.519819 18865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:55.520134 18865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:55.520169 18865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.520192 18865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:55.520205 18865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.525856 18865 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:43941
I20260923 22:20:55.525925 18993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:43941 every 8 connection(s)
I20260923 22:20:55.526266 18865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:20:55.531961 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 18865
I20260923 22:20:55.532091 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:20:55.532308 1541 external_mini_cluster.cc:1428] Setting key ea15424affb4920bd5b051c5e3a968c7
Sep 23 22:20:55 dist-test-slave-dmw7 krb5kdc[18490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:55.536324 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.527035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:54919 (local address 127.1.129.126:40533)
0923 22:20:55.527096 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.527098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.527303 (+ 205us) server_negotiation.cc:407] Connection header received
0923 22:20:55.527802 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.527804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.527815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.527878 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:55.528850 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.529388 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.530363 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.530563 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.530704 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.533280 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:55.533286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:55.533286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:55.533292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:55.534923 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.535355 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.535357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.535357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.535425 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.535679 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.535680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.535681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.535798 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:55.535876 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.536156 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.536232 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:20:55.536538 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.526975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:40533 (local address 127.1.129.67:54919)
0923 22:20:55.527155 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.527197 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:55.527685 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:55.527961 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:55.527964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:55.528521 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:55.528727 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.528731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.529477 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.529478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.530269 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.530270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.530707 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.530708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.530815 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.531311 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:55.531319 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:55.533192 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:55.535029 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:55.535036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:55.535038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:55.535262 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:55.535493 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:55.535494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:55.535495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:55.535610 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:55.535867 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:55.535873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:55.536022 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 22:20:55.536133 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 22:20:55.536292 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260923 22:20:55.537058 18994 heartbeater.cc:347] Connected to a master server at 127.1.129.126:40533
I20260923 22:20:55.537141 18994 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:55.537278 18994 heartbeater.cc:511] Master 127.1.129.126:40533 requested a full tablet report, sending...
I20260923 22:20:55.537626 18535 ts_manager.cc:194] Registered new tserver with Master: 9c68f99f4c134f68bb82d354cc5f983d (127.1.129.67:43941)
I20260923 22:20:55.538003 18535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:54919
I20260923 22:20:55.544049 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:20:55.551627 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.545378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.1.129.126:40533)
0923 22:20:55.545458 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.545461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.545529 (+ 68us) server_negotiation.cc:407] Connection header received
0923 22:20:55.545608 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.545609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.545621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.545688 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:55.546266 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.546814 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.547551 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.547739 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.547827 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.549176 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:55.549178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:55.549178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:55.549185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:55.550418 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.550788 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.550788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.550789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.550835 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:55.551053 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:55.551054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:55.551054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:55.551218 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:55.551305 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.551391 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.551500 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 22:20:55.553288 18535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:20:55.554874 18535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:20:55.561497 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.557461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.1.129.65:39725)
0923 22:20:55.557643 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.557647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.557685 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:20:55.557728 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.557731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.557748 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.557824 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:55.558156 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.559303 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.560570 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.561054 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.561149 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.561208 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.561362 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.561410 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260923 22:20:55.561525 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.557252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39725 (local address 127.0.0.1:51998)
0923 22:20:55.557513 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.557559 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:55.557623 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:55.558034 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:55.558035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:55.558037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:55.558093 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.558095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.559434 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.559435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.560472 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.560474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.561153 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.561155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.561244 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.561255 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 22:20:55.561343 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 22:20:55.561420 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":49,"thread_start_us":59,"threads_started":1}
I20260923 22:20:55.561542 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.557252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:43941 (local address 127.0.0.1:33234)
0923 22:20:55.557671 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.557681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:55.557739 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:55.557955 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:55.557958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:55.557960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:55.558025 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.558026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.559030 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.559031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.560184 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.560186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.561218 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.561219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.561281 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.561286 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:20:55.561335 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:20:55.561421 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260923 22:20:55.561643 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.557262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:45869 (local address 127.0.0.1:44044)
0923 22:20:55.557839 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.557848 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:55.557891 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:55.558174 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:55.558174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:55.558177 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:55.558209 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.558210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.559470 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.559471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.560499 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.560502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.561424 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.561424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.561481 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.561485 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:20:55.561533 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:20:55.561572 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":81,"threads_started":1}
I20260923 22:20:55.561792 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.557424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.1.129.67:43941)
0923 22:20:55.557615 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.557622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.557736 (+ 114us) server_negotiation.cc:407] Connection header received
0923 22:20:55.557839 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.557842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.557859 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.557941 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:55.558109 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.558926 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.560282 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.561052 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.561123 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.561182 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.561651 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.561705 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260923 22:20:55.561954 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.557484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.1.129.66:45869)
0923 22:20:55.557689 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.557694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.557963 (+ 269us) server_negotiation.cc:407] Connection header received
0923 22:20:55.558006 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.558008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.558023 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.558093 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:55.558291 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.559380 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.560597 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.561281 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.561343 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.561391 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.561802 (+ 411us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.561844 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260923 22:20:55.562686 18794 tablet_service.cc:1467] Processing CreateTablet for tablet 9297915ba2844b548e8d8767f9cce6d9 (DEFAULT_TABLE table=test-table [id=1733315dce74488c98c1a75d3f641688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:55.562681 18928 tablet_service.cc:1467] Processing CreateTablet for tablet 9297915ba2844b548e8d8767f9cce6d9 (DEFAULT_TABLE table=test-table [id=1733315dce74488c98c1a75d3f641688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:55.563050 18794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9297915ba2844b548e8d8767f9cce6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:55.563050 18928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9297915ba2844b548e8d8767f9cce6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:55.566421 19012 tablet_bootstrap.cc:492] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Bootstrap starting.
I20260923 22:20:55.566798 19011 tablet_bootstrap.cc:492] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: Bootstrap starting.
I20260923 22:20:55.567049 18660 tablet_service.cc:1467] Processing CreateTablet for tablet 9297915ba2844b548e8d8767f9cce6d9 (DEFAULT_TABLE table=test-table [id=1733315dce74488c98c1a75d3f641688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:55.567234 19012 tablet_bootstrap.cc:654] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:55.567371 18660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9297915ba2844b548e8d8767f9cce6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:55.567407 19011 tablet_bootstrap.cc:654] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:55.568037 19011 log.cc:824] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:55.568053 19012 log.cc:824] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:55.568893 19012 tablet_bootstrap.cc:492] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: No bootstrap required, opened a new log
I20260923 22:20:55.568998 19012 ts_tablet_manager.cc:1397] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 22:20:55.569036 19011 tablet_bootstrap.cc:492] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: No bootstrap required, opened a new log
I20260923 22:20:55.569106 19011 ts_tablet_manager.cc:1397] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:55.570343 19012 raft_consensus.cc:348] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.570561 19012 raft_consensus.cc:374] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:55.570608 19012 raft_consensus.cc:729] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c84d2026554ba8b97a56ae9b944d4b, State: Initialized, Role: FOLLOWER
I20260923 22:20:55.570539 19015 tablet_bootstrap.cc:492] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90: Bootstrap starting.
I20260923 22:20:55.570768 19012 consensus_queue.cc:260] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.571077 18860 heartbeater.cc:503] Master 127.1.129.126:40533 was elected leader, sending a full tablet report...
I20260923 22:20:55.571084 19012 ts_tablet_manager.cc:1428] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:55.571213 19011 raft_consensus.cc:348] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.571355 19015 tablet_bootstrap.cc:654] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:55.571383 19011 raft_consensus.cc:374] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:55.571435 19011 raft_consensus.cc:729] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c68f99f4c134f68bb82d354cc5f983d, State: Initialized, Role: FOLLOWER
I20260923 22:20:55.571586 19011 consensus_queue.cc:260] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.571878 18994 heartbeater.cc:503] Master 127.1.129.126:40533 was elected leader, sending a full tablet report...
I20260923 22:20:55.571938 19015 log.cc:824] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:55.571867 19011 ts_tablet_manager.cc:1428] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:20:55.572701 19015 tablet_bootstrap.cc:492] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90: No bootstrap required, opened a new log
I20260923 22:20:55.572789 19015 ts_tablet_manager.cc:1397] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:55.574182 19015 raft_consensus.cc:348] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.574333 19015 raft_consensus.cc:374] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:55.574378 19015 raft_consensus.cc:729] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d5dd0275242399bdb29e6241bce90, State: Initialized, Role: FOLLOWER
I20260923 22:20:55.574491 19015 consensus_queue.cc:260] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.574767 19015 ts_tablet_manager.cc:1428] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:20:55.574793 18726 heartbeater.cc:503] Master 127.1.129.126:40533 was elected leader, sending a full tablet report...
W20260923 22:20:55.626801 18861 tablet.cc:2369] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:55.645088 19016 raft_consensus.cc:482] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:55.645345 19016 raft_consensus.cc:504] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.645761 19016 leader_election.cc:288] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d5dd0275242399bdb29e6241bce90 (127.1.129.65:39725), 9c68f99f4c134f68bb82d354cc5f983d (127.1.129.67:43941)
I20260923 22:20:55.648840 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.645969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:43941 (local address 127.1.129.66:54739)
0923 22:20:55.646168 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.646177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:55.646223 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:55.646388 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:55.646389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:55.646391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:55.646454 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.646456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.647123 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.647124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.647868 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.647869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.648621 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.648622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.648685 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.648688 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:55.648737 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:20:55.648763 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":116,"threads_started":1}
I20260923 22:20:55.648859 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.646059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:54739 (local address 127.1.129.67:43941)
0923 22:20:55.646110 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.646112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.646265 (+ 153us) server_negotiation.cc:407] Connection header received
0923 22:20:55.646296 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.646298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.646307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.646363 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:55.646525 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.647008 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.647980 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.648495 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.648584 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.648592 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.648747 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.648790 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:20:55.648907 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.646035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:49419 (local address 127.1.129.65:39725)
0923 22:20:55.646083 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.646085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.646094 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:20:55.646122 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.646123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.646133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.646193 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:55.646389 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.646992 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.647981 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.648483 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.648556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.648566 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.648794 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.648837 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:20:55.649039 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.645942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39725 (local address 127.1.129.66:49419)
0923 22:20:55.645986 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.646001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:55.646048 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:55.646227 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:55.646229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:55.646231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:55.646310 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.646312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.647073 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.647074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.647866 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:55.647868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.648560 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.648561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:55.648643 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.648646 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:55.648697 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:20:55.648742 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":38}
I20260923 22:20:55.649281 18948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9297915ba2844b548e8d8767f9cce6d9" candidate_uuid: "31c84d2026554ba8b97a56ae9b944d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c68f99f4c134f68bb82d354cc5f983d" is_pre_election: true
I20260923 22:20:55.649364 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9297915ba2844b548e8d8767f9cce6d9" candidate_uuid: "31c84d2026554ba8b97a56ae9b944d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d5dd0275242399bdb29e6241bce90" is_pre_election: true
I20260923 22:20:55.649480 18948 raft_consensus.cc:2454] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c84d2026554ba8b97a56ae9b944d4b in term 0.
I20260923 22:20:55.649516 18680 raft_consensus.cc:2454] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c84d2026554ba8b97a56ae9b944d4b in term 0.
I20260923 22:20:55.649696 18749 leader_election.cc:302] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c84d2026554ba8b97a56ae9b944d4b, 9c68f99f4c134f68bb82d354cc5f983d; no voters:
I20260923 22:20:55.649844 19016 raft_consensus.cc:2781] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:55.649901 19016 raft_consensus.cc:482] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:55.649931 19016 raft_consensus.cc:3037] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:55.650664 19016 raft_consensus.cc:504] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.650805 19016 leader_election.cc:288] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [CANDIDATE]: Term 1 election: Requested vote from peers af0d5dd0275242399bdb29e6241bce90 (127.1.129.65:39725), 9c68f99f4c134f68bb82d354cc5f983d (127.1.129.67:43941)
I20260923 22:20:55.650908 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9297915ba2844b548e8d8767f9cce6d9" candidate_uuid: "31c84d2026554ba8b97a56ae9b944d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d5dd0275242399bdb29e6241bce90"
I20260923 22:20:55.650975 18948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9297915ba2844b548e8d8767f9cce6d9" candidate_uuid: "31c84d2026554ba8b97a56ae9b944d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c68f99f4c134f68bb82d354cc5f983d"
I20260923 22:20:55.650988 18680 raft_consensus.cc:3037] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:55.651047 18948 raft_consensus.cc:3037] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:55.651685 18680 raft_consensus.cc:2454] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c84d2026554ba8b97a56ae9b944d4b in term 1.
I20260923 22:20:55.651742 18948 raft_consensus.cc:2454] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c84d2026554ba8b97a56ae9b944d4b in term 1.
I20260923 22:20:55.651891 18750 leader_election.cc:302] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c84d2026554ba8b97a56ae9b944d4b, af0d5dd0275242399bdb29e6241bce90; no voters:
I20260923 22:20:55.652019 19016 raft_consensus.cc:2781] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:55.652156 19016 raft_consensus.cc:686] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 1 LEADER]: Becoming Leader. State: Replica: 31c84d2026554ba8b97a56ae9b944d4b, State: Running, Role: LEADER
I20260923 22:20:55.652253 19016 consensus_queue.cc:237] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } }
I20260923 22:20:55.653020 18535 catalog_manager.cc:5495] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31c84d2026554ba8b97a56ae9b944d4b (127.1.129.66). New cstate: current_term: 1 leader_uuid: "31c84d2026554ba8b97a56ae9b944d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31c84d2026554ba8b97a56ae9b944d4b" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 45869 } health_report { overall_health: HEALTHY } } }
I20260923 22:20:55.676056 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.673500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.1.129.66:45869)
0923 22:20:55.673548 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.673550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.673605 (+ 55us) server_negotiation.cc:407] Connection header received
0923 22:20:55.673742 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.673743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.673765 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.673852 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:55.674007 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.674600 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.675379 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.675553 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.675602 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.675734 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:55.675820 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:55.675878 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.675906 (+ 28us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.675930 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:20:55.677662 18948 raft_consensus.cc:1261] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 1 FOLLOWER]: Refusing update from remote peer 31c84d2026554ba8b97a56ae9b944d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:55.677662 18680 raft_consensus.cc:1261] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [term 1 FOLLOWER]: Refusing update from remote peer 31c84d2026554ba8b97a56ae9b944d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:55.678020 19016 consensus_queue.cc:1035] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c68f99f4c134f68bb82d354cc5f983d" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 43941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:55.678143 19021 consensus_queue.cc:1035] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d5dd0275242399bdb29e6241bce90" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 39725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:55.682435 18534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:55.682540 18534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:55.683331 18534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ef4614feba54d689b3a7bcda2f4d7d0: Sending DeleteTablet for 3 replicas of tablet 9297915ba2844b548e8d8767f9cce6d9
I20260923 22:20:55.683647 18794 tablet_service.cc:1514] Processing DeleteTablet for tablet 9297915ba2844b548e8d8767f9cce6d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:55 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:44044
I20260923 22:20:55.683676 18928 tablet_service.cc:1514] Processing DeleteTablet for tablet 9297915ba2844b548e8d8767f9cce6d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:55 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:33234
I20260923 22:20:55.683868 19034 tablet_replica.cc:330] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: stopping tablet replica
I20260923 22:20:55.683899 18660 tablet_service.cc:1514] Processing DeleteTablet for tablet 9297915ba2844b548e8d8767f9cce6d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:55 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:51998
I20260923 22:20:55.684028 19034 raft_consensus.cc:2229] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:55.684099 19036 tablet_replica.cc:330] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90: stopping tablet replica
I20260923 22:20:55.684165 19034 raft_consensus.cc:2258] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:55.684195 19036 raft_consensus.cc:2229] T 9297915ba2844b548e8d8767f9cce6d9 P af0d5dd0275242399bdb29e6241bce90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:55.684358 19035 tablet_replica.cc:330] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: stopping tablet replica
I20260923 22:20:55.684463 19035 raft_consensus.cc:2229] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:20:55.684559 19035 raft_consensus.cc:2258] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:55.684566 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 18597
I20260923 22:20:55.684749 19034 ts_tablet_manager.cc:1905] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:55.685362 19035 ts_tablet_manager.cc:1905] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:55.686947 19034 ts_tablet_manager.cc:1918] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:55.686977 19035 ts_tablet_manager.cc:1918] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:55.687024 19034 log.cc:1196] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/9297915ba2844b548e8d8767f9cce6d9
I20260923 22:20:55.687093 19035 log.cc:1196] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/9297915ba2844b548e8d8767f9cce6d9
I20260923 22:20:55.687305 19034 ts_tablet_manager.cc:1939] T 9297915ba2844b548e8d8767f9cce6d9 P 9c68f99f4c134f68bb82d354cc5f983d: Deleting consensus metadata
I20260923 22:20:55.687413 19035 ts_tablet_manager.cc:1939] T 9297915ba2844b548e8d8767f9cce6d9 P 31c84d2026554ba8b97a56ae9b944d4b: Deleting consensus metadata
I20260923 22:20:55.687695 18521 catalog_manager.cc:4841] TS 9c68f99f4c134f68bb82d354cc5f983d (127.1.129.67:43941): tablet 9297915ba2844b548e8d8767f9cce6d9 (table test-table [id=1733315dce74488c98c1a75d3f641688]) successfully deleted
I20260923 22:20:55.687856 18520 catalog_manager.cc:4841] TS 31c84d2026554ba8b97a56ae9b944d4b (127.1.129.66:45869): tablet 9297915ba2844b548e8d8767f9cce6d9 (table test-table [id=1733315dce74488c98c1a75d3f641688]) successfully deleted
W20260923 22:20:55.691567 18522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:39725 (error 108)
I20260923 22:20:55.691921 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 18731
I20260923 22:20:55.692054 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.691909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:39725 (local address 127.0.0.1:52004)
0923 22:20:55.691971 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.692020 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 22:20:55.693706 18522 catalog_manager.cc:4584] TS af0d5dd0275242399bdb29e6241bce90 (127.1.129.65:39725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9297915ba2844b548e8d8767f9cce6d9: Network error: Client connection negotiation failed: client connection to 127.1.129.65:39725: connect: Connection refused (error 111)
I20260923 22:20:55.697031 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 18865
I20260923 22:20:55.701354 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 18503
2026-09-23T22:20:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1939 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T22:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:20:55Z Disabled control of system clock
I20260923 22:20:55.727650 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:46745
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:46745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:20:55.807569 19041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:55.807772 19041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:55.807792 19041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:55.809012 19041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:55.809062 19041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:55.809084 19041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:55.809116 19041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:55.809144 19041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:55.810492 19041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:46745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:46745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:55.810770 19041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:55.811065 19041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:55.815680 19049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:55.815758 19041 server_base.cc:1034] running on GCE node
W20260923 22:20:55.815649 19047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:55.815814 19046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:55.816148 19041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:55.816453 19041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:55.817613 19041 hybrid_clock.cc:648] HybridClock initialized: now 1790202055817605 us; error 38 us; skew 500 ppm
I20260923 22:20:55.819027 19041 webserver.cc:460] Webserver started at http://127.1.129.126:45433/ using document root <none> and password file <none>
I20260923 22:20:55.819293 19041 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:55.819344 19041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:55.819540 19041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:55.820592 19041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "dee0d5b33d534c99838b7dafc22986b3"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "d3337d3ccbc329e855a0847bcf726a7e"
server_key_iv: "08c138d706c0e9975eec448b45695535"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.820960 19041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "dee0d5b33d534c99838b7dafc22986b3"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "d3337d3ccbc329e855a0847bcf726a7e"
server_key_iv: "08c138d706c0e9975eec448b45695535"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.822247 19041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:20:55.823063 19055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.823266 19041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:55.823386 19041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "dee0d5b33d534c99838b7dafc22986b3"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "d3337d3ccbc329e855a0847bcf726a7e"
server_key_iv: "08c138d706c0e9975eec448b45695535"
server_key_version: "encryptionkey@0"
I20260923 22:20:55.823468 19041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:55.833223 19041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:55.833549 19041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:55.833710 19041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:55.837716 19041 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:46745
I20260923 22:20:55.837774 19107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:46745 every 8 connection(s)
I20260923 22:20:55.838140 19041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:55.839177 19108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:55.841487 19108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3: Bootstrap starting.
I20260923 22:20:55.841596 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 19041
I20260923 22:20:55.841699 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:20:55.841912 1541 external_mini_cluster.cc:1428] Setting key f9195716e1e903c27f8aae51e5584054
I20260923 22:20:55.842216 19108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:55.842828 19108 log.cc:824] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:55.843583 19108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3: No bootstrap required, opened a new log
I20260923 22:20:55.845660 19108 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee0d5b33d534c99838b7dafc22986b3" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46745 } }
I20260923 22:20:55.845861 19108 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:55.845911 19108 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee0d5b33d534c99838b7dafc22986b3, State: Initialized, Role: FOLLOWER
I20260923 22:20:55.846040 19108 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee0d5b33d534c99838b7dafc22986b3" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46745 } }
I20260923 22:20:55.846117 19108 raft_consensus.cc:388] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:55.846159 19108 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:55.846207 19108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:55.846993 19108 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee0d5b33d534c99838b7dafc22986b3" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46745 } }
I20260923 22:20:55.847107 19108 leader_election.cc:302] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee0d5b33d534c99838b7dafc22986b3; no voters:
I20260923 22:20:55.847312 19108 leader_election.cc:288] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:20:55.847343 19113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:55.847443 19113 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [term 1 LEADER]: Becoming Leader. State: Replica: dee0d5b33d534c99838b7dafc22986b3, State: Running, Role: LEADER
I20260923 22:20:55.847561 19113 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee0d5b33d534c99838b7dafc22986b3" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46745 } }
I20260923 22:20:55.847877 19113 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee0d5b33d534c99838b7dafc22986b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee0d5b33d534c99838b7dafc22986b3" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46745 } } }
I20260923 22:20:55.847971 19113 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:55.848245 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.842405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.1.129.126:46745)
0923 22:20:55.842618 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:55.842663 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:55.842697 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:20:55.843172 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:55.843179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:55.843237 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:55.843352 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:55.843576 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.844535 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.845752 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:55.846314 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:55.846410 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:55.847074 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:55.847077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:55.847078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:55.847302 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:55.847335 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:55.847346 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:55.847646 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:55.847683 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 22:20:55.847771 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260923 22:20:55.848206 19114 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee0d5b33d534c99838b7dafc22986b3. Latest consensus state: current_term: 1 leader_uuid: "dee0d5b33d534c99838b7dafc22986b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee0d5b33d534c99838b7dafc22986b3" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 46745 } } }
I20260923 22:20:55.848327 19114 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:55.848496 19108 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:55.849309 19119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:55.850133 19119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:55.852311 19119 catalog_manager.cc:1347] Generated new cluster ID: 536898683e6c4635a75a952dddaa8c61
I20260923 22:20:55.852448 19119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:20:55.870239 19119 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:20:55.870878 19119 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:55.880002 19119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3: Generated new TSK 0
I20260923 22:20:55.880196 19119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:20:55.886396 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46745
--builtin_ntp_servers=127.1.129.84:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:20:55.988054 19132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:55.988252 19132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:55.988273 19132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:55.989418 19132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:55.989459 19132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:55.989485 19132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:20:55.990860 19132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:55.991098 19132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:55.991433 19132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:55.997155 19137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:55.997287 19140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:55.997155 19138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:55.997447 19132 server_base.cc:1034] running on GCE node
I20260923 22:20:55.997659 19132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:55.997910 19132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:55.999074 19132 hybrid_clock.cc:648] HybridClock initialized: now 1790202055999062 us; error 33 us; skew 500 ppm
I20260923 22:20:56.000371 19132 webserver.cc:460] Webserver started at http://127.1.129.65:41747/ using document root <none> and password file <none>
I20260923 22:20:56.000629 19132 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:56.000697 19132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:56.000842 19132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:56.001958 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "2b77e00369d94bf1927f1762ff99bcd9"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "6323d8ec9cf455bb5ea4bc6f21012e2c"
server_key_iv: "dfa2bd8453bdf019c29a73dfe2e65ec3"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.002342 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "2b77e00369d94bf1927f1762ff99bcd9"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "6323d8ec9cf455bb5ea4bc6f21012e2c"
server_key_iv: "dfa2bd8453bdf019c29a73dfe2e65ec3"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.003675 19132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:20:56.004374 19146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.004539 19132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:20:56.004642 19132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "2b77e00369d94bf1927f1762ff99bcd9"
format_stamp: "Formatted at 2026-09-23 22:20:55 on dist-test-slave-dmw7"
server_key: "6323d8ec9cf455bb5ea4bc6f21012e2c"
server_key_iv: "dfa2bd8453bdf019c29a73dfe2e65ec3"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.004714 19132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:56.040583 19132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:56.040936 19132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:56.041096 19132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:56.041344 19132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:56.041679 19132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:56.041735 19132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.041778 19132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:56.041807 19132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.047513 19132 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:45203
I20260923 22:20:56.047574 19259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:45203 every 8 connection(s)
I20260923 22:20:56.047938 19132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:56.052172 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 19132
I20260923 22:20:56.052294 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:20:56.052490 1541 external_mini_cluster.cc:1428] Setting key 4909f2c6b6de7f91748e96450b2b0406
I20260923 22:20:56.053526 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46745
--builtin_ntp_servers=127.1.129.84:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 22:20:56.055012 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.048754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:52167 (local address 127.1.129.126:46745)
0923 22:20:56.048820 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.048822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.049065 (+ 243us) server_negotiation.cc:407] Connection header received
0923 22:20:56.049628 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.049629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.049642 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.049700 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:56.050401 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.051005 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.051991 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.052205 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.052288 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.054732 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:56.054733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:56.054734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:56.054738 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:56.054752 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:56.054756 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:56.054805 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.054930 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.054952 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:20:56.055306 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.048770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46745 (local address 127.1.129.65:52167)
0923 22:20:56.048940 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.048974 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:56.049492 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:56.049791 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:56.049794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:56.049802 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:56.050303 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.050307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.051116 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.051122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.051894 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.051896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.052298 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.052299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.052459 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.053137 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:20:56.053138 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:56.053177 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:20:56.053207 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:56.054831 (+ 1624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:56.054839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:56.054842 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:56.054929 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:20:56.055052 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260923 22:20:56.055873 19260 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46745
I20260923 22:20:56.055975 19260 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:56.056165 19260 heartbeater.cc:511] Master 127.1.129.126:46745 requested a full tablet report, sending...
I20260923 22:20:56.056675 19072 ts_manager.cc:194] Registered new tserver with Master: 2b77e00369d94bf1927f1762ff99bcd9 (127.1.129.65:45203)
I20260923 22:20:56.057372 19072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.65:52167
W20260923 22:20:56.136231 19263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:56.136422 19263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:56.136443 19263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:56.137570 19263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:56.137609 19263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:56.137658 19263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:20:56.139024 19263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:56.139274 19263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:56.139555 19263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:56.144783 19268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:56.144833 19269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:56.144865 19263 server_base.cc:1034] running on GCE node
W20260923 22:20:56.144840 19271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:56.145354 19263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:56.145596 19263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:56.146700 19263 hybrid_clock.cc:648] HybridClock initialized: now 1790202056146700 us; error 43 us; skew 500 ppm
I20260923 22:20:56.147867 19263 webserver.cc:460] Webserver started at http://127.1.129.66:40291/ using document root <none> and password file <none>
I20260923 22:20:56.148146 19263 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:56.148216 19263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:56.148370 19263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:56.149415 19263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "1c0ddc3882bf45a79d300ffefbaab5b7"
format_stamp: "Formatted at 2026-09-23 22:20:56 on dist-test-slave-dmw7"
server_key: "da56665344479d7a18af7109814b8ea8"
server_key_iv: "2afda7746a2bb665d91060730ba67e1b"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.149775 19263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "1c0ddc3882bf45a79d300ffefbaab5b7"
format_stamp: "Formatted at 2026-09-23 22:20:56 on dist-test-slave-dmw7"
server_key: "da56665344479d7a18af7109814b8ea8"
server_key_iv: "2afda7746a2bb665d91060730ba67e1b"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.151096 19263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:20:56.151832 19277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.152030 19263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:56.152129 19263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "1c0ddc3882bf45a79d300ffefbaab5b7"
format_stamp: "Formatted at 2026-09-23 22:20:56 on dist-test-slave-dmw7"
server_key: "da56665344479d7a18af7109814b8ea8"
server_key_iv: "2afda7746a2bb665d91060730ba67e1b"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.152208 19263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:56.176600 19263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:56.176911 19263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:56.177059 19263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:56.177286 19263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:56.177640 19263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:56.177695 19263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.177736 19263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:56.177764 19263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.183696 19263 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:35413
I20260923 22:20:56.183738 19390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:35413 every 8 connection(s)
I20260923 22:20:56.184047 19263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:20:56.188709 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 19263
I20260923 22:20:56.188844 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:20:56.189056 1541 external_mini_cluster.cc:1428] Setting key f07c4c796e6db75032855b23ab61a482
I20260923 22:20:56.189074 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.184749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:50041 (local address 127.1.129.126:46745)
0923 22:20:56.184833 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.184835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.185037 (+ 202us) server_negotiation.cc:407] Connection header received
0923 22:20:56.185553 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.185555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.185567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.185622 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:56.186269 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.186783 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.187727 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.187930 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.187986 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.188797 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:56.188799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:56.188799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:56.188804 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:56.188820 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:56.188825 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:56.188877 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.188988 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.189018 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 22:20:56.189340 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.184713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46745 (local address 127.1.129.66:50041)
0923 22:20:56.184914 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.184947 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:56.185416 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:56.185684 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:56.185688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:56.185696 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:56.186185 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.186189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.186861 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.186862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.187648 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.187650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.188003 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.188004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.188115 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.188657 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:20:56.188657 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:56.188685 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:20:56.188706 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:56.188887 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:56.188894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:56.188897 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:56.188994 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 22:20:56.189105 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260923 22:20:56.189791 19391 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46745
I20260923 22:20:56.189885 19391 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:56.190037 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46745
--builtin_ntp_servers=127.1.129.84:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 22:20:56.190094 19391 heartbeater.cc:511] Master 127.1.129.126:46745 requested a full tablet report, sending...
I20260923 22:20:56.190457 19072 ts_manager.cc:194] Registered new tserver with Master: 1c0ddc3882bf45a79d300ffefbaab5b7 (127.1.129.66:35413)
I20260923 22:20:56.190903 19072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.66:50041
W20260923 22:20:56.269783 19394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:56.269975 19394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:56.269996 19394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:56.271157 19394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:56.271193 19394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:56.271217 19394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:20:56.272523 19394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:46745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:56.272724 19394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:56.272984 19394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:56.277779 19399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:56.277832 19402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:56.277961 19394 server_base.cc:1034] running on GCE node
W20260923 22:20:56.277782 19400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:56.278237 19394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:56.278512 19394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:56.279675 19394 hybrid_clock.cc:648] HybridClock initialized: now 1790202056279662 us; error 25 us; skew 500 ppm
I20260923 22:20:56.280843 19394 webserver.cc:460] Webserver started at http://127.1.129.67:45083/ using document root <none> and password file <none>
I20260923 22:20:56.281071 19394 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:56.281141 19394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:56.281293 19394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:56.282285 19394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "7e865213b5f740b690c9f5ac94332557"
format_stamp: "Formatted at 2026-09-23 22:20:56 on dist-test-slave-dmw7"
server_key: "83004c0f13ea41cac70ceda35a281eeb"
server_key_iv: "ac4e2f1ecbff122b38ce953a804c48d3"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.282676 19394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "7e865213b5f740b690c9f5ac94332557"
format_stamp: "Formatted at 2026-09-23 22:20:56 on dist-test-slave-dmw7"
server_key: "83004c0f13ea41cac70ceda35a281eeb"
server_key_iv: "ac4e2f1ecbff122b38ce953a804c48d3"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.283954 19394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:56.284749 19408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.284916 19394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:56.285010 19394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "7e865213b5f740b690c9f5ac94332557"
format_stamp: "Formatted at 2026-09-23 22:20:56 on dist-test-slave-dmw7"
server_key: "83004c0f13ea41cac70ceda35a281eeb"
server_key_iv: "ac4e2f1ecbff122b38ce953a804c48d3"
server_key_version: "encryptionkey@0"
I20260923 22:20:56.285094 19394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:56.316826 19394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:56.317140 19394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:56.317291 19394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:56.317540 19394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:56.317879 19394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:56.317934 19394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.317976 19394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:56.318004 19394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:56.323921 19394 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:40199
I20260923 22:20:56.323979 19521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:40199 every 8 connection(s)
I20260923 22:20:56.324323 19394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:20:56.325093 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 19394
I20260923 22:20:56.325198 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:20:56.325379 1541 external_mini_cluster.cc:1428] Setting key a92a662539c06be0ed26c789700234c1
I20260923 22:20:56.329798 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.325098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:40329 (local address 127.1.129.126:46745)
0923 22:20:56.325172 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.325175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.325431 (+ 256us) server_negotiation.cc:407] Connection header received
0923 22:20:56.326137 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.326138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.326152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.326227 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:56.327062 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.327662 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.328585 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.328768 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.328825 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.329562 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:56.329563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:56.329563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:56.329567 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:56.329580 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:56.329584 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:56.329629 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.329722 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.329742 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:20:56.330039 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.325078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:46745 (local address 127.1.129.67:40329)
0923 22:20:56.325287 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.325327 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:56.326033 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:56.326297 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:56.326303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:56.326317 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:56.326933 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.326939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.327739 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.327740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.328501 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.328503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.328843 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.328844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.328948 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.329436 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:20:56.329437 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:56.329464 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:20:56.329485 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:56.329644 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:56.329647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:56.329649 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:20:56.329706 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 22:20:56.329782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
I20260923 22:20:56.330683 19522 heartbeater.cc:347] Connected to a master server at 127.1.129.126:46745
I20260923 22:20:56.330791 19522 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:56.330963 19522 heartbeater.cc:511] Master 127.1.129.126:46745 requested a full tablet report, sending...
I20260923 22:20:56.331327 19072 ts_manager.cc:194] Registered new tserver with Master: 7e865213b5f740b690c9f5ac94332557 (127.1.129.67:40199)
I20260923 22:20:56.331707 19072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.67:40329
I20260923 22:20:56.336952 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:20:56.341363 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.338333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.1.129.126:46745)
0923 22:20:56.338386 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.338388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.338453 (+ 65us) server_negotiation.cc:407] Connection header received
0923 22:20:56.338489 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.338489 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.338497 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.338547 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:56.338788 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.339285 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.340045 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.340221 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.340275 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.341045 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:56.341046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:56.341047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:56.341051 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:56.341067 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:56.341071 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:56.341139 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.341287 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.341308 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:20:56.343091 19072 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:20:56.344692 19072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:20:56.351377 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.347289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.1.129.65:45203)
0923 22:20:56.347480 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.347483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.347491 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:56.347528 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.347530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.347545 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.347631 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:56.347878 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.348980 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.350281 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.350978 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.351080 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.351185 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.351254 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.351292 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260923 22:20:56.351377 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.347076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:45203 (local address 127.0.0.1:57028)
0923 22:20:56.347274 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.347281 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:56.347313 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:56.347758 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:56.347759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:56.347761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:56.347797 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.347798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.349084 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.349085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.350196 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.350198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.351083 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.351083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.351149 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.351169 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 22:20:56.351238 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 22:20:56.351283 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":11,"thread_start_us":61,"threads_started":1}
I20260923 22:20:56.351490 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.346882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:40199 (local address 127.0.0.1:34726)
0923 22:20:56.347111 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.347121 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:56.347171 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:56.347614 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:56.347614 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:56.347616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:56.347669 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.347671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.348834 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.348835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.350148 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.350151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.351209 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.351211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.351262 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.351265 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:56.351293 (+ 28us) client_negotiation.cc:240] Negotiation successful
0923 22:20:56.351331 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":61,"threads_started":1}
I20260923 22:20:56.351604 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.346846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35413 (local address 127.0.0.1:49584)
0923 22:20:56.347034 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.347076 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:56.347139 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:56.347504 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:56.347508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:56.347510 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:56.347576 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.347579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.349188 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.349189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.350085 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.350087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.351055 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.351056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.351141 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.351159 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 22:20:56.351208 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:20:56.351260 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260923 22:20:56.351627 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.346940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.1.129.66:35413)
0923 22:20:56.347312 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.347317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.347328 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:20:56.347368 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.347371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.347388 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.347493 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:56.347794 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.349091 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.350196 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.350949 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.351333 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.351432 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.351458 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.351507 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":80,"threads_started":1}
I20260923 22:20:56.352200 19194 tablet_service.cc:1467] Processing CreateTablet for tablet b4747f7e669848458ded9af8e2b06756 (DEFAULT_TABLE table=test-table [id=acd836db19ec41d09b7b1f47502149f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:56.352296 19325 tablet_service.cc:1467] Processing CreateTablet for tablet b4747f7e669848458ded9af8e2b06756 (DEFAULT_TABLE table=test-table [id=acd836db19ec41d09b7b1f47502149f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:56.352567 19194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4747f7e669848458ded9af8e2b06756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:56.352648 19325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4747f7e669848458ded9af8e2b06756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:56.354969 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.346986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.1.129.67:40199)
0923 22:20:56.347433 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.347438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.347448 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:20:56.347488 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.347491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.347507 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.347741 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:56.347799 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.348731 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.350233 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.351118 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.351417 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.354807 (+ 3390us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.354836 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.354882 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":109,"threads_started":1}
I20260923 22:20:56.355419 19537 tablet_bootstrap.cc:492] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9: Bootstrap starting.
I20260923 22:20:56.355659 19536 tablet_bootstrap.cc:492] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Bootstrap starting.
I20260923 22:20:56.355749 19456 tablet_service.cc:1467] Processing CreateTablet for tablet b4747f7e669848458ded9af8e2b06756 (DEFAULT_TABLE table=test-table [id=acd836db19ec41d09b7b1f47502149f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:56.356081 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4747f7e669848458ded9af8e2b06756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:56.356101 19537 tablet_bootstrap.cc:654] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:56.356215 19536 tablet_bootstrap.cc:654] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:56.356396 19537 log.cc:824] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:56.356426 19536 log.cc:824] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:56.357182 19536 tablet_bootstrap.cc:492] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: No bootstrap required, opened a new log
I20260923 22:20:56.357240 19536 ts_tablet_manager.cc:1397] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:20:56.359516 19540 tablet_bootstrap.cc:492] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: Bootstrap starting.
I20260923 22:20:56.359819 19536 raft_consensus.cc:348] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.359946 19536 raft_consensus.cc:374] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:56.359967 19536 raft_consensus.cc:729] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0ddc3882bf45a79d300ffefbaab5b7, State: Initialized, Role: FOLLOWER
I20260923 22:20:56.360101 19540 tablet_bootstrap.cc:654] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:56.360109 19536 consensus_queue.cc:260] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.360380 19536 ts_tablet_manager.cc:1428] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:20:56.360417 19540 log.cc:824] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:56.360435 19391 heartbeater.cc:503] Master 127.1.129.126:46745 was elected leader, sending a full tablet report...
I20260923 22:20:56.360864 19537 tablet_bootstrap.cc:492] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9: No bootstrap required, opened a new log
I20260923 22:20:56.360944 19537 ts_tablet_manager.cc:1397] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260923 22:20:56.361232 19540 tablet_bootstrap.cc:492] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: No bootstrap required, opened a new log
I20260923 22:20:56.361344 19540 ts_tablet_manager.cc:1397] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:56.363590 19537 raft_consensus.cc:348] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.363777 19537 raft_consensus.cc:374] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:56.363814 19537 raft_consensus.cc:729] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b77e00369d94bf1927f1762ff99bcd9, State: Initialized, Role: FOLLOWER
I20260923 22:20:56.363945 19537 consensus_queue.cc:260] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.363970 19540 raft_consensus.cc:348] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.364120 19540 raft_consensus.cc:374] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:56.364167 19540 raft_consensus.cc:729] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e865213b5f740b690c9f5ac94332557, State: Initialized, Role: FOLLOWER
I20260923 22:20:56.364259 19537 ts_tablet_manager.cc:1428] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 22:20:56.364293 19260 heartbeater.cc:503] Master 127.1.129.126:46745 was elected leader, sending a full tablet report...
I20260923 22:20:56.364286 19540 consensus_queue.cc:260] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.364573 19540 ts_tablet_manager.cc:1428] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:20:56.364609 19522 heartbeater.cc:503] Master 127.1.129.126:46745 was elected leader, sending a full tablet report...
W20260923 22:20:56.434597 19392 tablet.cc:2369] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:56.516736 19541 raft_consensus.cc:482] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:56.516856 19541 raft_consensus.cc:504] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.517150 19541 leader_election.cc:288] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e865213b5f740b690c9f5ac94332557 (127.1.129.67:40199), 2b77e00369d94bf1927f1762ff99bcd9 (127.1.129.65:45203)
I20260923 22:20:56.521064 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.517635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:34031 (local address 127.1.129.65:45203)
0923 22:20:56.517700 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.517702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.517710 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:56.517770 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.517771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.517780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.517811 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:56.517953 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.518617 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.520123 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.520691 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.520781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.520790 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.520953 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.520991 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:20:56.521064 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.517511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:40359 (local address 127.1.129.67:40199)
0923 22:20:56.517575 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.517577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.517587 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:20:56.517619 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.517620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.517629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.517692 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:56.517857 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.518617 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.520123 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.520691 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.520781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.520790 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.520951 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.520991 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:20:56.521070 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.517381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:40199 (local address 127.1.129.66:40359)
0923 22:20:56.517462 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.517474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:56.517555 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:56.517712 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:56.517713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:56.517715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:56.517785 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.517787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.518717 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.518718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.520007 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.520010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.520787 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.520789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.520867 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.520871 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:20:56.520929 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:20:56.520979 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":81}
I20260923 22:20:56.521076 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.517484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:45203 (local address 127.1.129.66:34031)
0923 22:20:56.517640 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.517650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:56.517712 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:56.517830 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:56.517830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:56.517832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:56.517889 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.517890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.518720 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.518720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.520007 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:56.520010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.520789 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.520789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:56.520871 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.520874 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:56.520929 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:20:56.520979 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":57,"thread_start_us":96,"threads_started":1}
I20260923 22:20:56.521512 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4747f7e669848458ded9af8e2b06756" candidate_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e865213b5f740b690c9f5ac94332557" is_pre_election: true
I20260923 22:20:56.521627 19476 raft_consensus.cc:2454] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0ddc3882bf45a79d300ffefbaab5b7 in term 0.
I20260923 22:20:56.521829 19279 leader_election.cc:302] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0ddc3882bf45a79d300ffefbaab5b7, 7e865213b5f740b690c9f5ac94332557; no voters:
I20260923 22:20:56.521927 19214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4747f7e669848458ded9af8e2b06756" candidate_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b77e00369d94bf1927f1762ff99bcd9" is_pre_election: true
I20260923 22:20:56.521971 19541 raft_consensus.cc:2781] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:56.522071 19541 raft_consensus.cc:482] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:56.522097 19214 raft_consensus.cc:2454] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0ddc3882bf45a79d300ffefbaab5b7 in term 0.
I20260923 22:20:56.522107 19541 raft_consensus.cc:3037] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:56.523020 19541 raft_consensus.cc:504] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.523168 19541 leader_election.cc:288] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [CANDIDATE]: Term 1 election: Requested vote from peers 7e865213b5f740b690c9f5ac94332557 (127.1.129.67:40199), 2b77e00369d94bf1927f1762ff99bcd9 (127.1.129.65:45203)
I20260923 22:20:56.523303 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4747f7e669848458ded9af8e2b06756" candidate_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e865213b5f740b690c9f5ac94332557"
I20260923 22:20:56.523360 19214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4747f7e669848458ded9af8e2b06756" candidate_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b77e00369d94bf1927f1762ff99bcd9"
I20260923 22:20:56.523398 19476 raft_consensus.cc:3037] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:56.523469 19214 raft_consensus.cc:3037] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:56.524118 19476 raft_consensus.cc:2454] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0ddc3882bf45a79d300ffefbaab5b7 in term 1.
I20260923 22:20:56.524327 19279 leader_election.cc:302] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0ddc3882bf45a79d300ffefbaab5b7, 7e865213b5f740b690c9f5ac94332557; no voters:
I20260923 22:20:56.524327 19214 raft_consensus.cc:2454] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0ddc3882bf45a79d300ffefbaab5b7 in term 1.
I20260923 22:20:56.524438 19541 raft_consensus.cc:2781] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:56.524617 19541 raft_consensus.cc:686] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0ddc3882bf45a79d300ffefbaab5b7, State: Running, Role: LEADER
I20260923 22:20:56.524735 19541 consensus_queue.cc:237] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } }
I20260923 22:20:56.525565 19072 catalog_manager.cc:5495] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0ddc3882bf45a79d300ffefbaab5b7 (127.1.129.66). New cstate: current_term: 1 leader_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ddc3882bf45a79d300ffefbaab5b7" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 } health_report { overall_health: UNKNOWN } } }
I20260923 22:20:56.531857 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.529751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.1.129.66:35413)
0923 22:20:56.529803 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:56.529805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:56.529813 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:56.529848 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:56.529848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:56.529862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:56.529931 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:56.530028 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.530462 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.531219 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:56.531411 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:56.531477 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:56.531590 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:56.531677 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:56.531735 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:56.531766 (+ 31us) server_negotiation.cc:299] Negotiation successful
0923 22:20:56.531781 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:20:56.533751 19214 raft_consensus.cc:1261] T b4747f7e669848458ded9af8e2b06756 P 2b77e00369d94bf1927f1762ff99bcd9 [term 1 FOLLOWER]: Refusing update from remote peer 1c0ddc3882bf45a79d300ffefbaab5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:56.533797 19476 raft_consensus.cc:1261] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 1 FOLLOWER]: Refusing update from remote peer 1c0ddc3882bf45a79d300ffefbaab5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:56.534221 19541 consensus_queue.cc:1035] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e865213b5f740b690c9f5ac94332557" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:56.534346 19546 consensus_queue.cc:1035] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b77e00369d94bf1927f1762ff99bcd9" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:56.536624 19549 mvcc.cc:205] Tried to move back new op lower bound from 7332667623558979584 to 7332667623525904384. Current Snapshot: MvccSnapshot[applied={T|T < 7332667623558979584}]
I20260923 22:20:56.536855 19551 mvcc.cc:205] Tried to move back new op lower bound from 7332667623558979584 to 7332667623525904384. Current Snapshot: MvccSnapshot[applied={T|T < 7332667623558979584}]
I20260923 22:20:56.539577 19071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:56.539670 19071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:56.540446 19071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee0d5b33d534c99838b7dafc22986b3: Sending DeleteTablet for 3 replicas of tablet b4747f7e669848458ded9af8e2b06756
I20260923 22:20:56.540738 19325 tablet_service.cc:1514] Processing DeleteTablet for tablet b4747f7e669848458ded9af8e2b06756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:56 UTC) from {username='slave'} at 127.0.0.1:49584
I20260923 22:20:56.540766 19194 tablet_service.cc:1514] Processing DeleteTablet for tablet b4747f7e669848458ded9af8e2b06756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:56 UTC) from {username='slave'} at 127.0.0.1:57028
I20260923 22:20:56.540786 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet b4747f7e669848458ded9af8e2b06756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:56 UTC) from {username='slave'} at 127.0.0.1:34726
I20260923 22:20:56.540982 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 19132
I20260923 22:20:56.541599 19558 tablet_replica.cc:330] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: stopping tablet replica
I20260923 22:20:56.541703 19558 raft_consensus.cc:2229] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:20:56.541926 19559 tablet_replica.cc:330] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: stopping tablet replica
I20260923 22:20:56.542052 19559 raft_consensus.cc:2229] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:56.542191 19559 raft_consensus.cc:2258] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:56.541816 19558 raft_consensus.cc:2258] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:56.542830 19558 ts_tablet_manager.cc:1905] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:56.543144 19559 ts_tablet_manager.cc:1905] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:56.545181 19559 ts_tablet_manager.cc:1918] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:56.545291 19559 log.cc:1196] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/b4747f7e669848458ded9af8e2b06756
I20260923 22:20:56.545601 19559 ts_tablet_manager.cc:1939] T b4747f7e669848458ded9af8e2b06756 P 7e865213b5f740b690c9f5ac94332557: Deleting consensus metadata
I20260923 22:20:56.546053 19057 catalog_manager.cc:4841] TS 7e865213b5f740b690c9f5ac94332557 (127.1.129.67:40199): tablet b4747f7e669848458ded9af8e2b06756 (table test-table [id=acd836db19ec41d09b7b1f47502149f0]) successfully deleted
I20260923 22:20:56.547006 19558 ts_tablet_manager.cc:1918] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:56.547079 19558 log.cc:1196] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/b4747f7e669848458ded9af8e2b06756
I20260923 22:20:56.547380 19558 ts_tablet_manager.cc:1939] T b4747f7e669848458ded9af8e2b06756 P 1c0ddc3882bf45a79d300ffefbaab5b7: Deleting consensus metadata
I20260923 22:20:56.547771 19058 catalog_manager.cc:4841] TS 1c0ddc3882bf45a79d300ffefbaab5b7 (127.1.129.66:35413): tablet b4747f7e669848458ded9af8e2b06756 (table test-table [id=acd836db19ec41d09b7b1f47502149f0]) successfully deleted
W20260923 22:20:56.548475 19057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:45203 (error 108)
I20260923 22:20:56.548699 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 19263
I20260923 22:20:56.549278 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.548815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:45203 (local address 127.0.0.1:57044)
0923 22:20:56.549203 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.549243 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:45203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369}
W20260923 22:20:56.549611 19057 catalog_manager.cc:4584] TS 2b77e00369d94bf1927f1762ff99bcd9 (127.1.129.65:45203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4747f7e669848458ded9af8e2b06756: Network error: Client connection negotiation failed: client connection to 127.1.129.65:45203: connect: Connection refused (error 111)
I20260923 22:20:56.553820 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 19394
I20260923 22:20:56.558681 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 19041
2026-09-23T22:20:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (857 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:20:56 dist-test-slave-dmw7 krb5kdc[19564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:20:56 dist-test-slave-dmw7 krb5kdc[19564](info): set up 2 sockets
Sep 23 22:20:56 dist-test-slave-dmw7 krb5kdc[19564](info): commencing operation
krb5kdc: starting...
W20260923 22:20:57.590396 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:20:57 dist-test-slave-dmw7 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:20:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:20:57.705626 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36283
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:57.785032 19577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:57.785254 19577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:57.785292 19577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:57.786415 19577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:57.786480 19577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:57.786517 19577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:57.786562 19577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:57.786592 19577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:57.788034 19577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:57.788286 19577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:57.788506 19577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:57.794174 19583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:57.794201 19585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:57.794180 19577 server_base.cc:1034] running on GCE node
W20260923 22:20:57.794183 19582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:57.794710 19577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:57.794951 19577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:57.796085 19577 hybrid_clock.cc:648] HybridClock initialized: now 1790202057796072 us; error 29 us; skew 500 ppm
Sep 23 22:20:57 dist-test-slave-dmw7 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:57.798477 19577 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:20:57.798990 19577 webserver.cc:460] Webserver started at http://127.1.129.126:34173/ using document root <none> and password file <none>
I20260923 22:20:57.799227 19577 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:57.799273 19577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:57.799418 19577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:57.800457 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "421a0bf0804e44afb92d0c0bd5ef98eb"
format_stamp: "Formatted at 2026-09-23 22:20:57 on dist-test-slave-dmw7"
server_key: "1f574b312060948ccc40b979f1d85ed5"
server_key_iv: "226c3dc0c9d3a0cf73522416ee581d9f"
server_key_version: "encryptionkey@0"
I20260923 22:20:57.800827 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "421a0bf0804e44afb92d0c0bd5ef98eb"
format_stamp: "Formatted at 2026-09-23 22:20:57 on dist-test-slave-dmw7"
server_key: "1f574b312060948ccc40b979f1d85ed5"
server_key_iv: "226c3dc0c9d3a0cf73522416ee581d9f"
server_key_version: "encryptionkey@0"
I20260923 22:20:57.802137 19577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:20:57.802898 19592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:57.803098 19577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:57.803184 19577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "421a0bf0804e44afb92d0c0bd5ef98eb"
format_stamp: "Formatted at 2026-09-23 22:20:57 on dist-test-slave-dmw7"
server_key: "1f574b312060948ccc40b979f1d85ed5"
server_key_iv: "226c3dc0c9d3a0cf73522416ee581d9f"
server_key_version: "encryptionkey@0"
I20260923 22:20:57.803236 19577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:57.817903 19577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:57.819751 19577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:57.819854 19577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:57.823397 19577 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:36283
I20260923 22:20:57.823426 19644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:36283 every 8 connection(s)
I20260923 22:20:57.823734 19577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:57.824558 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:57.826577 19645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb: Bootstrap starting.
I20260923 22:20:57.827107 19645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:57.827314 19645 log.cc:824] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:57.827941 19645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb: No bootstrap required, opened a new log
I20260923 22:20:57.829385 19645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421a0bf0804e44afb92d0c0bd5ef98eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36283 } }
I20260923 22:20:57.829507 19645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:57.829528 19645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421a0bf0804e44afb92d0c0bd5ef98eb, State: Initialized, Role: FOLLOWER
I20260923 22:20:57.829650 19645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421a0bf0804e44afb92d0c0bd5ef98eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36283 } }
I20260923 22:20:57.829728 19645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:57.829751 19645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:57.829777 19645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:57.829980 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 19577
I20260923 22:20:57.830056 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:20:57.830250 1541 external_mini_cluster.cc:1428] Setting key 357d611b0a4abea6e66a9353dbf274ff
I20260923 22:20:57.830394 19645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421a0bf0804e44afb92d0c0bd5ef98eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36283 } }
I20260923 22:20:57.830498 19645 leader_election.cc:302] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 421a0bf0804e44afb92d0c0bd5ef98eb; no voters:
I20260923 22:20:57.830696 19645 leader_election.cc:288] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:20:57.830725 19648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:57.830930 19648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [term 1 LEADER]: Becoming Leader. State: Replica: 421a0bf0804e44afb92d0c0bd5ef98eb, State: Running, Role: LEADER
I20260923 22:20:57.830936 19645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:57.831133 19648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421a0bf0804e44afb92d0c0bd5ef98eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36283 } }
I20260923 22:20:57.831627 19652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "421a0bf0804e44afb92d0c0bd5ef98eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421a0bf0804e44afb92d0c0bd5ef98eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36283 } } }
I20260923 22:20:57.831707 19652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:57.831756 19651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 421a0bf0804e44afb92d0c0bd5ef98eb. Latest consensus state: current_term: 1 leader_uuid: "421a0bf0804e44afb92d0c0bd5ef98eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421a0bf0804e44afb92d0c0bd5ef98eb" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36283 } } }
I20260923 22:20:57.831858 19651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb [sys.catalog]: This master's current role is: LEADER
W20260923 22:20:57.832887 19666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:20:57.832942 19666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:20:57.832998 19659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:57.833127 19659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:57.834601 19659 catalog_manager.cc:1347] Generated new cluster ID: 04b17af1bbc34e0a9ed46424695d13a9
I20260923 22:20:57.834671 19659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:20:57 dist-test-slave-dmw7 krb5kdc[19564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:57.843016 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:57.830698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.1.129.126:36283)
0923 22:20:57.830861 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:57.830872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:57.830983 (+ 111us) server_negotiation.cc:407] Connection header received
0923 22:20:57.831292 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:57.831299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:57.831353 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:57.831466 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:57.832814 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:57.834069 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:57.835193 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:57.836767 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:57.836855 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:57.839402 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:57.839406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:57.839406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:57.839416 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:57.841237 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:57.841586 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:57.841588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:57.841589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:57.841673 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:57.841841 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:57.841841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:57.841842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:57.842106 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:57.842231 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:57.842686 (+ 455us) server_negotiation.cc:299] Negotiation successful
0923 22:20:57.842807 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260923 22:20:57.845865 19659 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:20:57.846313 19659 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:57.852766 19659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb: Generated new TSK 0
I20260923 22:20:57.852947 19659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:20:57.884974 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36283
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:57.981719 19671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:57.981916 19671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:57.981940 19671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:57.983142 19671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:57.983192 19671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:57.983230 19671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:20:57.984678 19671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:57.984885 19671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:57.985126 19671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:57.990034 19676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:57.990036 19677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:57.990044 19679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:57.990221 19671 server_base.cc:1034] running on GCE node
I20260923 22:20:57.990422 19671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:57.990609 19671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:57.991894 19671 hybrid_clock.cc:648] HybridClock initialized: now 1790202057991869 us; error 28 us; skew 500 ppm
Sep 23 22:20:57 dist-test-slave-dmw7 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:57.994364 19671 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:20:57.994884 19671 webserver.cc:460] Webserver started at http://127.1.129.65:46721/ using document root <none> and password file <none>
I20260923 22:20:57.995108 19671 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:57.995151 19671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:57.995296 19671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:57.996308 19671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "f872e5ac9bf24812a89dfbf4a2cff416"
format_stamp: "Formatted at 2026-09-23 22:20:57 on dist-test-slave-dmw7"
server_key: "18b0e5dfa3780d79a182bc4cb996f33f"
server_key_iv: "6bbbebf8731a4d80c2169df93891a6c5"
server_key_version: "encryptionkey@0"
I20260923 22:20:57.996663 19671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "f872e5ac9bf24812a89dfbf4a2cff416"
format_stamp: "Formatted at 2026-09-23 22:20:57 on dist-test-slave-dmw7"
server_key: "18b0e5dfa3780d79a182bc4cb996f33f"
server_key_iv: "6bbbebf8731a4d80c2169df93891a6c5"
server_key_version: "encryptionkey@0"
I20260923 22:20:57.997934 19671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:20:57.998616 19686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:57.998818 19671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:57.998929 19671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "f872e5ac9bf24812a89dfbf4a2cff416"
format_stamp: "Formatted at 2026-09-23 22:20:57 on dist-test-slave-dmw7"
server_key: "18b0e5dfa3780d79a182bc4cb996f33f"
server_key_iv: "6bbbebf8731a4d80c2169df93891a6c5"
server_key_version: "encryptionkey@0"
I20260923 22:20:57.998997 19671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:58.011878 19671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:58.013825 19671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:58.014000 19671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:58.014241 19671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:58.014557 19671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:58.014590 19671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.014635 19671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:58.014748 19671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.020509 19671 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:45959
I20260923 22:20:58.020563 19799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:45959 every 8 connection(s)
I20260923 22:20:58.020862 19671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:20:58 dist-test-slave-dmw7 krb5kdc[19564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:58.030333 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 19671
I20260923 22:20:58.030360 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.021612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:53853 (local address 127.1.129.126:36283)
0923 22:20:58.021706 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.021708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.021897 (+ 189us) server_negotiation.cc:407] Connection header received
0923 22:20:58.022441 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.022442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.022458 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.022512 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.023532 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.024051 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.025067 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.025289 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.025368 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.027580 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:58.027581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:58.027582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:58.027589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:58.028952 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.029373 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.029374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.029375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.029435 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.029682 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.029683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.029683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.029804 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:58.029874 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.030158 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.030257 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:20:58.030476 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:20:58.030552 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.021559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36283 (local address 127.1.129.65:53853)
0923 22:20:58.021760 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.021808 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.022329 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.022580 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.022584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.023214 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:58.023408 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.023413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.024154 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.024156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.024966 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.024968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.025376 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.025378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.025511 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.026063 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:58.026071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:58.027478 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:58.029039 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:58.029045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:58.029048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:58.029281 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:58.029502 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:58.029503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:58.029503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:58.029612 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:58.029876 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:58.029883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:58.030022 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.030132 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.030293 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260923 22:20:58.030741 1541 external_mini_cluster.cc:1428] Setting key 329acff5895227538ba8966693bcd915
I20260923 22:20:58.031351 19800 heartbeater.cc:347] Connected to a master server at 127.1.129.126:36283
I20260923 22:20:58.031459 19800 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:58.031643 19800 heartbeater.cc:511] Master 127.1.129.126:36283 requested a full tablet report, sending...
I20260923 22:20:58.032235 19609 ts_manager.cc:194] Registered new tserver with Master: f872e5ac9bf24812a89dfbf4a2cff416 (127.1.129.65:45959)
I20260923 22:20:58.032927 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:53853
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:20:58.055334 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36283
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:58.135471 19805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:58.135694 19805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:58.135721 19805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:58.136835 19805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:58.136873 19805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:58.136898 19805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:20:58.138288 19805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:58.138468 19805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:58.138664 19805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:58.143615 19810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.143616 19813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.143657 19811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:58.143975 19805 server_base.cc:1034] running on GCE node
I20260923 22:20:58.144209 19805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:58.144459 19805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:58.145630 19805 hybrid_clock.cc:648] HybridClock initialized: now 1790202058145622 us; error 42 us; skew 500 ppm
Sep 23 22:20:58 dist-test-slave-dmw7 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:58.148172 19805 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:20:58.148591 19805 webserver.cc:460] Webserver started at http://127.1.129.66:38419/ using document root <none> and password file <none>
I20260923 22:20:58.148818 19805 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:58.148890 19805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:58.149047 19805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:58.150115 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "c56dfb298f4d4416ace926dd72e4ad4a"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f31a919719cdad9b86093d72e7cc4c99"
server_key_iv: "e0fdf031e0643737843da6314e877bc7"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.150480 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "c56dfb298f4d4416ace926dd72e4ad4a"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f31a919719cdad9b86093d72e7cc4c99"
server_key_iv: "e0fdf031e0643737843da6314e877bc7"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.151865 19805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:58.152639 19820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.152818 19805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:58.152915 19805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "c56dfb298f4d4416ace926dd72e4ad4a"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f31a919719cdad9b86093d72e7cc4c99"
server_key_iv: "e0fdf031e0643737843da6314e877bc7"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.152998 19805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:58.168686 19805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:58.170593 19805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:58.170888 19805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:58.171115 19805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:58.171422 19805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:58.171474 19805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.171514 19805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:58.171541 19805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.177095 19805 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:35757
I20260923 22:20:58.177203 19933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:35757 every 8 connection(s)
I20260923 22:20:58.177488 19805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:20:58.180012 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 19805
I20260923 22:20:58.180097 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:20:58.180315 1541 external_mini_cluster.cc:1428] Setting key d930bbbd33e787b1ac231758cde666b3
Sep 23 22:20:58 dist-test-slave-dmw7 krb5kdc[19564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:58.187763 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.178295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:50275 (local address 127.1.129.126:36283)
0923 22:20:58.178368 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.178370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.178565 (+ 195us) server_negotiation.cc:407] Connection header received
0923 22:20:58.179237 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.179238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.179248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.179300 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.180281 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.180993 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.181959 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.182270 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.182344 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.184777 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:58.184779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:58.184779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:58.184787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:58.186253 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.186794 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.186795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.186796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.186860 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.187114 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.187114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.187115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.187231 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:58.187307 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.187578 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.187677 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:20:58.187943 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.178245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36283 (local address 127.1.129.66:50275)
0923 22:20:58.178440 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.178478 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.179143 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.179355 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.179358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.179945 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:58.180204 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.180210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.181091 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.181092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.181871 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.181873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.182364 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.182365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.182514 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.183059 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:58.183069 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:58.184686 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:58.186355 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:58.186363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:58.186365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:58.186710 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:58.186926 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:58.186927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:58.186927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:58.187040 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:58.187298 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:58.187304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:58.187444 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.187546 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.187701 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260923 22:20:58.188473 19934 heartbeater.cc:347] Connected to a master server at 127.1.129.126:36283
I20260923 22:20:58.188589 19934 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:58.188807 19934 heartbeater.cc:511] Master 127.1.129.126:36283 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
I20260923 22:20:58.189226 19609 ts_manager.cc:194] Registered new tserver with Master: c56dfb298f4d4416ace926dd72e4ad4a (127.1.129.66:35757)
I20260923 22:20:58.189710 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:50275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:20:58.205885 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36283
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:58.286352 19939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:58.286563 19939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:58.286586 19939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:58.287736 19939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:58.287777 19939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:58.287803 19939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:20:58.289242 19939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:58.289419 19939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:58.289673 19939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:58.294673 19944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.294613 19947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:58.294682 19939 server_base.cc:1034] running on GCE node
W20260923 22:20:58.294621 19945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:58.295081 19939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:58.295313 19939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:58.296458 19939 hybrid_clock.cc:648] HybridClock initialized: now 1790202058296444 us; error 27 us; skew 500 ppm
Sep 23 22:20:58 dist-test-slave-dmw7 krb5kdc[19564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:58.298852 19939 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:20:58.299311 19939 webserver.cc:460] Webserver started at http://127.1.129.67:35983/ using document root <none> and password file <none>
I20260923 22:20:58.299531 19939 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:58.299575 19939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:58.299713 19939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:58.300706 19939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "f7f1971eb1f04087965d8e93873c0e13"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f8f9ca12d5188082061aee92f15ea9a6"
server_key_iv: "a07f8537e235fb4e6ffd308fddfd8b83"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.301065 19939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "f7f1971eb1f04087965d8e93873c0e13"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f8f9ca12d5188082061aee92f15ea9a6"
server_key_iv: "a07f8537e235fb4e6ffd308fddfd8b83"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.302459 19939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:20:58.303413 19954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.303646 19939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:58.303719 19939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "f7f1971eb1f04087965d8e93873c0e13"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f8f9ca12d5188082061aee92f15ea9a6"
server_key_iv: "a07f8537e235fb4e6ffd308fddfd8b83"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.303812 19939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:58.345096 19939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:58.347402 19939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:58.347584 19939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:58.347849 19939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:58.348217 19939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:58.348275 19939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.348320 19939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:58.348347 19939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.354780 19939 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:38973
I20260923 22:20:58.354831 20067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:38973 every 8 connection(s)
I20260923 22:20:58.355199 19939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:20:58.360855 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 19939
I20260923 22:20:58.360936 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:20:58.361156 1541 external_mini_cluster.cc:1428] Setting key d2d3e038ff32aaa82c30c4b8db74838c
Sep 23 22:20:58 dist-test-slave-dmw7 krb5kdc[19564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:20:58.364902 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.356072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:53695 (local address 127.1.129.126:36283)
0923 22:20:58.356157 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.356160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.356341 (+ 181us) server_negotiation.cc:407] Connection header received
0923 22:20:58.356924 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.356926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.356936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.357004 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.357927 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.358429 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.359506 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.359699 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.359761 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.362266 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:58.362267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:58.362267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:58.362273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:58.363734 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.364098 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.364099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.364099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.364180 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.364388 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.364388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.364389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.364491 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:58.364541 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.364777 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.364838 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:20:58.365077 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.356025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:36283 (local address 127.1.129.67:53695)
0923 22:20:58.356209 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.356249 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.356806 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.357071 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.357075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.357653 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:58.357850 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.357854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.358524 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.358525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.359426 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.359429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.359810 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.359811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.359922 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.360440 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:58.360449 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:58.362183 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:58.363799 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:58.363805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:58.363806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:58.364029 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:58.364237 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:58.364237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:58.364238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:58.364334 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:58.364544 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:58.364548 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:58.364667 (+ 119us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.364749 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.364867 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260923 22:20:58.365537 20068 heartbeater.cc:347] Connected to a master server at 127.1.129.126:36283
I20260923 22:20:58.365650 20068 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:58.365828 20068 heartbeater.cc:511] Master 127.1.129.126:36283 requested a full tablet report, sending...
I20260923 22:20:58.366221 19609 ts_manager.cc:194] Registered new tserver with Master: f7f1971eb1f04087965d8e93873c0e13 (127.1.129.67:38973)
I20260923 22:20:58.366593 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:53695
I20260923 22:20:58.373031 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:20:58.380152 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.374392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.1.129.126:36283)
0923 22:20:58.374465 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.374467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.374513 (+ 46us) server_negotiation.cc:407] Connection header received
0923 22:20:58.374580 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.374581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.374590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.374678 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.375199 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.375672 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.376359 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.376534 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.376590 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.377860 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:58.377861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:58.377861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:58.377866 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:58.379096 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.379468 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.379468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.379469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.379520 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.379681 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.379682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.379682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.379800 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:58.379877 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.379974 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.380021 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 22:20:58.381929 19609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:20:58.383648 19609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:20:58.390359 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.386012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:38973 (local address 127.0.0.1:47960)
0923 22:20:58.386185 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.386227 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.386294 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.386561 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.386564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.386567 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:58.386667 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.386670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.388098 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.388099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.389181 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.389183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.390108 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.390109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.390157 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.390168 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.390217 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.390270 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":4,"thread_start_us":125,"threads_started":1}
I20260923 22:20:58.390518 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.386029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35757 (local address 127.0.0.1:40410)
0923 22:20:58.386879 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.386888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.386936 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.387201 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.387202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.387204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:58.387240 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.387241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.388399 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.388400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.389397 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.389399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.390255 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.390256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.390319 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.390324 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.390380 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.390423 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260923 22:20:58.390534 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.386202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.1.129.67:38973)
0923 22:20:58.386375 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.386380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.386391 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:20:58.386435 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.386437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.386450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.386556 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:58.386760 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.387991 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.389271 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.389860 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.390317 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.390385 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.390411 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.390460 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260923 22:20:58.390841 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.386225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.1.129.65:45959)
0923 22:20:58.386455 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.386461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.386653 (+ 192us) server_negotiation.cc:407] Connection header received
0923 22:20:58.386699 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.386702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.386716 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.386783 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:58.386935 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.387760 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.389291 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.389971 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.390050 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.390109 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.390473 (+ 364us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.390517 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260923 22:20:58.390975 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.386012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:45959 (local address 127.0.0.1:58184)
0923 22:20:58.386505 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.386515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.386566 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.386793 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.386794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.386796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:58.386853 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.386855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.387867 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.387868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.389167 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.389170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.390056 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.390057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.390152 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.390168 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.390237 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.390270 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260923 22:20:58.391331 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.386413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.1.129.66:35757)
0923 22:20:58.386737 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.386741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.386995 (+ 254us) server_negotiation.cc:407] Connection header received
0923 22:20:58.387038 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.387040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.387054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.387121 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:58.387322 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.388303 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.389483 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.390170 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.390252 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.390307 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.390394 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.391042 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":85,"threads_started":1}
I20260923 22:20:58.391281 20002 tablet_service.cc:1467] Processing CreateTablet for tablet b77f306f95f4426abef4cce33292ae6c (DEFAULT_TABLE table=test-table [id=ee46426e20bb48f290caf7789ae2a967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:58.391659 20002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77f306f95f4426abef4cce33292ae6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:58.391672 19734 tablet_service.cc:1467] Processing CreateTablet for tablet b77f306f95f4426abef4cce33292ae6c (DEFAULT_TABLE table=test-table [id=ee46426e20bb48f290caf7789ae2a967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:58.391880 19868 tablet_service.cc:1467] Processing CreateTablet for tablet b77f306f95f4426abef4cce33292ae6c (DEFAULT_TABLE table=test-table [id=ee46426e20bb48f290caf7789ae2a967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:58.392010 19734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77f306f95f4426abef4cce33292ae6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:58.392169 19868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77f306f95f4426abef4cce33292ae6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:58.394407 20085 tablet_bootstrap.cc:492] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: Bootstrap starting.
I20260923 22:20:58.395102 20085 tablet_bootstrap.cc:654] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:58.395545 20086 tablet_bootstrap.cc:492] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a: Bootstrap starting.
I20260923 22:20:58.395747 20085 log.cc:824] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:58.396214 20086 tablet_bootstrap.cc:654] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:58.396528 20085 tablet_bootstrap.cc:492] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: No bootstrap required, opened a new log
I20260923 22:20:58.396616 20085 ts_tablet_manager.cc:1397] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:58.396734 20086 log.cc:824] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:58.397070 20087 tablet_bootstrap.cc:492] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416: Bootstrap starting.
I20260923 22:20:58.397460 20086 tablet_bootstrap.cc:492] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a: No bootstrap required, opened a new log
I20260923 22:20:58.397523 20086 ts_tablet_manager.cc:1397] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:20:58.397722 20087 tablet_bootstrap.cc:654] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:58.398295 20087 log.cc:824] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:58.398409 20085 raft_consensus.cc:348] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.398623 20085 raft_consensus.cc:374] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:58.398693 20085 raft_consensus.cc:729] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1971eb1f04087965d8e93873c0e13, State: Initialized, Role: FOLLOWER
I20260923 22:20:58.398818 20085 consensus_queue.cc:260] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.399138 20085 ts_tablet_manager.cc:1428] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:20:58.399147 20087 tablet_bootstrap.cc:492] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416: No bootstrap required, opened a new log
I20260923 22:20:58.399220 20087 ts_tablet_manager.cc:1397] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:58.399284 20068 heartbeater.cc:503] Master 127.1.129.126:36283 was elected leader, sending a full tablet report...
I20260923 22:20:58.399744 20086 raft_consensus.cc:348] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.399916 20086 raft_consensus.cc:374] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:58.399952 20086 raft_consensus.cc:729] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c56dfb298f4d4416ace926dd72e4ad4a, State: Initialized, Role: FOLLOWER
I20260923 22:20:58.400104 20086 consensus_queue.cc:260] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.400391 20086 ts_tablet_manager.cc:1428] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:20:58.400419 19934 heartbeater.cc:503] Master 127.1.129.126:36283 was elected leader, sending a full tablet report...
I20260923 22:20:58.401397 20087 raft_consensus.cc:348] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.401566 20087 raft_consensus.cc:374] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:58.401592 20087 raft_consensus.cc:729] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f872e5ac9bf24812a89dfbf4a2cff416, State: Initialized, Role: FOLLOWER
I20260923 22:20:58.401702 20087 consensus_queue.cc:260] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.401952 20087 ts_tablet_manager.cc:1428] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:20:58.401974 19800 heartbeater.cc:503] Master 127.1.129.126:36283 was elected leader, sending a full tablet report...
I20260923 22:20:58.425127 20093 raft_consensus.cc:482] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:58.425271 20093 raft_consensus.cc:504] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.425638 20093 leader_election.cc:288] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7f1971eb1f04087965d8e93873c0e13 (127.1.129.67:38973), c56dfb298f4d4416ace926dd72e4ad4a (127.1.129.66:35757)
W20260923 22:20:58.428041 19935 tablet.cc:2369] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:58.428464 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.425808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35757 (local address 127.1.129.65:46845)
0923 22:20:58.425873 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.425888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.425943 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.426122 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.426123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.426125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:58.426206 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.426208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.426909 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.426910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.427655 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.427656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.428278 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.428279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.428339 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.428342 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.428384 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.428413 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":21}
I20260923 22:20:58.428539 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.425955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:46845 (local address 127.1.129.66:35757)
0923 22:20:58.426003 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.426005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.426013 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:58.426036 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.426037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.426053 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.426131 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:58.426280 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.426821 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.427747 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.428165 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.428233 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.428241 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.428419 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.428468 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:20:58.428792 19888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77f306f95f4426abef4cce33292ae6c" candidate_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" is_pre_election: true
I20260923 22:20:58.428934 19888 raft_consensus.cc:2454] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f872e5ac9bf24812a89dfbf4a2cff416 in term 0.
I20260923 22:20:58.429040 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.425808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:38973 (local address 127.1.129.65:47403)
0923 22:20:58.426034 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.426043 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.426124 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.426248 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.426248 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.426250 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:58.426320 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.426322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.427009 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.427010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.427750 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.427751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.428519 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.428520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.428860 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.428864 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.428934 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.428972 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":24,"thread_start_us":126,"threads_started":1}
I20260923 22:20:58.429057 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.425961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:47403 (local address 127.1.129.67:38973)
0923 22:20:58.426011 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.426013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.426111 (+ 98us) server_negotiation.cc:407] Connection header received
0923 22:20:58.426179 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.426180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.426188 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.426244 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:58.426387 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.426929 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.427847 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.428429 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.428500 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.428508 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.428956 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.428996 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:20:58.429175 19688 leader_election.cc:302] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c56dfb298f4d4416ace926dd72e4ad4a, f872e5ac9bf24812a89dfbf4a2cff416; no voters:
I20260923 22:20:58.429315 20093 raft_consensus.cc:2781] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:58.429358 20093 raft_consensus.cc:482] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:58.429399 20093 raft_consensus.cc:3037] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:58.429503 20022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77f306f95f4426abef4cce33292ae6c" candidate_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f1971eb1f04087965d8e93873c0e13" is_pre_election: true
I20260923 22:20:58.429644 20022 raft_consensus.cc:2454] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f872e5ac9bf24812a89dfbf4a2cff416 in term 0.
I20260923 22:20:58.430186 20093 raft_consensus.cc:504] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.430315 20093 leader_election.cc:288] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [CANDIDATE]: Term 1 election: Requested vote from peers f7f1971eb1f04087965d8e93873c0e13 (127.1.129.67:38973), c56dfb298f4d4416ace926dd72e4ad4a (127.1.129.66:35757)
I20260923 22:20:58.430440 20022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77f306f95f4426abef4cce33292ae6c" candidate_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f1971eb1f04087965d8e93873c0e13"
I20260923 22:20:58.430511 20022 raft_consensus.cc:3037] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:58.430482 19888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77f306f95f4426abef4cce33292ae6c" candidate_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c56dfb298f4d4416ace926dd72e4ad4a"
I20260923 22:20:58.430598 19888 raft_consensus.cc:3037] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:58.431332 20022 raft_consensus.cc:2454] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f872e5ac9bf24812a89dfbf4a2cff416 in term 1.
I20260923 22:20:58.431447 19888 raft_consensus.cc:2454] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f872e5ac9bf24812a89dfbf4a2cff416 in term 1.
I20260923 22:20:58.431505 19690 leader_election.cc:302] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f7f1971eb1f04087965d8e93873c0e13, f872e5ac9bf24812a89dfbf4a2cff416; no voters:
I20260923 22:20:58.431604 20093 raft_consensus.cc:2781] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:58.431797 20093 raft_consensus.cc:686] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [term 1 LEADER]: Becoming Leader. State: Replica: f872e5ac9bf24812a89dfbf4a2cff416, State: Running, Role: LEADER
I20260923 22:20:58.431898 20093 consensus_queue.cc:237] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } }
I20260923 22:20:58.432799 19609 catalog_manager.cc:5495] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 reported cstate change: term changed from 0 to 1, leader changed from <none> to f872e5ac9bf24812a89dfbf4a2cff416 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f872e5ac9bf24812a89dfbf4a2cff416" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 45959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 } health_report { overall_health: UNKNOWN } } }
I20260923 22:20:58.439105 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.436690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.1.129.65:45959)
0923 22:20:58.436743 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.436746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.436790 (+ 44us) server_negotiation.cc:407] Connection header received
0923 22:20:58.436862 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.436863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.436882 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.436964 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:58.437089 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.437538 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.438305 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.438499 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.438583 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.438765 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:58.438857 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:58.438964 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.439004 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.439016 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:20:58.440766 19888 raft_consensus.cc:1261] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 1 FOLLOWER]: Refusing update from remote peer f872e5ac9bf24812a89dfbf4a2cff416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:58.440766 20022 raft_consensus.cc:1261] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 1 FOLLOWER]: Refusing update from remote peer f872e5ac9bf24812a89dfbf4a2cff416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:58.441135 20093 consensus_queue.cc:1035] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c56dfb298f4d4416ace926dd72e4ad4a" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:58.441284 20095 consensus_queue.cc:1035] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f1971eb1f04087965d8e93873c0e13" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 38973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:58.443723 20100 mvcc.cc:205] Tried to move back new op lower bound from 7332667631370678272 to 7332667631337713664. Current Snapshot: MvccSnapshot[applied={T|T < 7332667631370678272}]
I20260923 22:20:58.445812 19608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:58.445900 19608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:58.446616 19608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 421a0bf0804e44afb92d0c0bd5ef98eb: Sending DeleteTablet for 3 replicas of tablet b77f306f95f4426abef4cce33292ae6c
I20260923 22:20:58.446950 19868 tablet_service.cc:1514] Processing DeleteTablet for tablet b77f306f95f4426abef4cce33292ae6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:58 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:40410
I20260923 22:20:58.446962 20002 tablet_service.cc:1514] Processing DeleteTablet for tablet b77f306f95f4426abef4cce33292ae6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:58 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:47960
I20260923 22:20:58.446970 19734 tablet_service.cc:1514] Processing DeleteTablet for tablet b77f306f95f4426abef4cce33292ae6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:58 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:58184
I20260923 22:20:58.447384 20110 tablet_replica.cc:330] T b77f306f95f4426abef4cce33292ae6c P f872e5ac9bf24812a89dfbf4a2cff416: stopping tablet replica
I20260923 22:20:58.447485 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 19671
W20260923 22:20:58.453047 19595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:45959 (error 108)
I20260923 22:20:58.453115 20108 tablet_replica.cc:330] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: stopping tablet replica
I20260923 22:20:58.453140 20109 tablet_replica.cc:330] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a: stopping tablet replica
I20260923 22:20:58.453219 20108 raft_consensus.cc:2229] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:58.453233 20109 raft_consensus.cc:2229] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:58.453333 20108 raft_consensus.cc:2258] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:58.453334 20109 raft_consensus.cc:2258] T b77f306f95f4426abef4cce33292ae6c P c56dfb298f4d4416ace926dd72e4ad4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:58.453495 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 19805
I20260923 22:20:58.453606 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.453494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:45959 (local address 127.0.0.1:58196)
0923 22:20:58.453537 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.453573 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 22:20:58.453795 20108 ts_tablet_manager.cc:1905] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 22:20:58.454156 19595 catalog_manager.cc:4584] TS f872e5ac9bf24812a89dfbf4a2cff416 (127.1.129.65:45959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b77f306f95f4426abef4cce33292ae6c: Network error: Client connection negotiation failed: client connection to 127.1.129.65:45959: connect: Connection refused (error 111)
I20260923 22:20:58.455960 20108 ts_tablet_manager.cc:1918] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:58.456089 20108 log.cc:1196] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/b77f306f95f4426abef4cce33292ae6c
I20260923 22:20:58.456336 20108 ts_tablet_manager.cc:1939] T b77f306f95f4426abef4cce33292ae6c P f7f1971eb1f04087965d8e93873c0e13: Deleting consensus metadata
I20260923 22:20:58.456720 19596 catalog_manager.cc:4841] TS f7f1971eb1f04087965d8e93873c0e13 (127.1.129.67:38973): tablet b77f306f95f4426abef4cce33292ae6c (table test-table [id=ee46426e20bb48f290caf7789ae2a967]) successfully deleted
I20260923 22:20:58.458178 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.458070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:35757 (local address 127.0.0.1:40424)
0923 22:20:58.458118 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.458147 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.66:35757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 22:20:58.458256 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 19939
I20260923 22:20:58.462735 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 19577
2026-09-23T22:20:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1904 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T22:20:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:20:58Z Disabled control of system clock
I20260923 22:20:58.489974 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:40747
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:20:58.568764 20115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:58.568975 20115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:58.568995 20115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:58.570171 20115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:58.570214 20115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:58.570227 20115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:58.570240 20115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:58.570295 20115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:58.571672 20115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:40747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:40747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:58.571928 20115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:58.572235 20115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:58.577052 20123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.577059 20121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.577051 20120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:58.577163 20115 server_base.cc:1034] running on GCE node
I20260923 22:20:58.577510 20115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:58.577798 20115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:58.578959 20115 hybrid_clock.cc:648] HybridClock initialized: now 1790202058578944 us; error 27 us; skew 500 ppm
I20260923 22:20:58.580266 20115 webserver.cc:460] Webserver started at http://127.1.129.126:34469/ using document root <none> and password file <none>
I20260923 22:20:58.580513 20115 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:58.580585 20115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:58.580741 20115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:58.581794 20115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "a58ba5449ff447b68c76770c6284ae19"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "28c262475dd80c014f27c2be21e4704a"
server_key_iv: "2664505acebe36109d6f32dc1ee82bf8"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.582158 20115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "a58ba5449ff447b68c76770c6284ae19"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "28c262475dd80c014f27c2be21e4704a"
server_key_iv: "2664505acebe36109d6f32dc1ee82bf8"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.583525 20115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:20:58.584362 20129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.584676 20115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:58.584781 20115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "a58ba5449ff447b68c76770c6284ae19"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "28c262475dd80c014f27c2be21e4704a"
server_key_iv: "2664505acebe36109d6f32dc1ee82bf8"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.584857 20115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:58.599928 20115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:58.600267 20115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:58.600423 20115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:58.604566 20115 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:40747
I20260923 22:20:58.604602 20181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:40747 every 8 connection(s)
I20260923 22:20:58.604946 20115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:58.605952 20182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:58.608263 20182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19: Bootstrap starting.
I20260923 22:20:58.608840 20182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:58.609138 20182 log.cc:824] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:58.609807 20182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19: No bootstrap required, opened a new log
I20260923 22:20:58.611380 20182 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ba5449ff447b68c76770c6284ae19" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40747 } }
I20260923 22:20:58.611512 20182 raft_consensus.cc:374] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:58.611534 20182 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58ba5449ff447b68c76770c6284ae19, State: Initialized, Role: FOLLOWER
I20260923 22:20:58.611730 20182 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ba5449ff447b68c76770c6284ae19" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40747 } }
I20260923 22:20:58.611814 20182 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:58.611864 20182 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:58.611914 20182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:58.612416 20182 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ba5449ff447b68c76770c6284ae19" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40747 } }
I20260923 22:20:58.612524 20182 leader_election.cc:302] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58ba5449ff447b68c76770c6284ae19; no voters:
I20260923 22:20:58.612704 20182 leader_election.cc:288] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:20:58.612744 20185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:58.612926 20185 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [term 1 LEADER]: Becoming Leader. State: Replica: a58ba5449ff447b68c76770c6284ae19, State: Running, Role: LEADER
I20260923 22:20:58.612960 20182 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:58.613034 20185 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ba5449ff447b68c76770c6284ae19" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40747 } }
I20260923 22:20:58.613401 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58ba5449ff447b68c76770c6284ae19. Latest consensus state: current_term: 1 leader_uuid: "a58ba5449ff447b68c76770c6284ae19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ba5449ff447b68c76770c6284ae19" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40747 } } }
I20260923 22:20:58.613503 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:58.613526 20186 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a58ba5449ff447b68c76770c6284ae19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ba5449ff447b68c76770c6284ae19" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40747 } } }
I20260923 22:20:58.613631 20186 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:58.613724 20194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:58.614200 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 20115
I20260923 22:20:58.614252 20194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:58.614308 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:20:58.614552 1541 external_mini_cluster.cc:1428] Setting key 02e8486d77f2262b650de8940bce5a60
I20260923 22:20:58.615921 20194 catalog_manager.cc:1347] Generated new cluster ID: 0cf7e5bea4e9413db1a36db93f8ff458
I20260923 22:20:58.615972 20194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:20:58.620381 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.615012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.1.129.126:40747)
0923 22:20:58.615192 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.615201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.615234 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:20:58.615346 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.615353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.615419 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.615583 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.615809 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.617160 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.618226 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.618885 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.619011 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.619682 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:58.619686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:58.619687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:58.619891 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:58.619930 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:58.619944 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:58.620013 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.620080 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.620158 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260923 22:20:58.662583 20194 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:20:58.663167 20194 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:58.668128 20194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19: Generated new TSK 0
I20260923 22:20:58.668331 20194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:20:58.672361 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40747
--builtin_ntp_servers=127.1.129.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:20:58.751726 20206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:58.751914 20206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:58.751935 20206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:58.753026 20206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:58.753064 20206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:58.753088 20206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:20:58.754377 20206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:58.754602 20206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:58.754904 20206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:58.759790 20211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.759804 20214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.759859 20212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:58.759882 20206 server_base.cc:1034] running on GCE node
I20260923 22:20:58.760154 20206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:58.760411 20206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:58.761581 20206 hybrid_clock.cc:648] HybridClock initialized: now 1790202058761565 us; error 31 us; skew 500 ppm
I20260923 22:20:58.765573 20206 webserver.cc:460] Webserver started at http://127.1.129.65:39659/ using document root <none> and password file <none>
I20260923 22:20:58.765820 20206 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:58.765866 20206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:58.766036 20206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:58.767172 20206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "eb408166028041ad998e785739efe291"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f5527b1b63052444da24565bbffe90b3"
server_key_iv: "0f02b137a0f6d65f40c8e4e383cb40ff"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.767513 20206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "eb408166028041ad998e785739efe291"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f5527b1b63052444da24565bbffe90b3"
server_key_iv: "0f02b137a0f6d65f40c8e4e383cb40ff"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.768733 20206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:20:58.769402 20220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.769570 20206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:58.769667 20206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "eb408166028041ad998e785739efe291"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "f5527b1b63052444da24565bbffe90b3"
server_key_iv: "0f02b137a0f6d65f40c8e4e383cb40ff"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.769742 20206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:58.785691 20206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:58.785987 20206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:58.786142 20206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:58.786381 20206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:58.786819 20206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:58.786875 20206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.786917 20206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:58.786943 20206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.792734 20206 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:44905
I20260923 22:20:58.792783 20333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:44905 every 8 connection(s)
I20260923 22:20:58.793111 20206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:58.796984 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 20206
I20260923 22:20:58.797096 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:20:58.797297 1541 external_mini_cluster.cc:1428] Setting key df785131492f0e6ef00e7c7195d4ba99
I20260923 22:20:58.798394 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.793791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:48243 (local address 127.1.129.126:40747)
0923 22:20:58.793865 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.793867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.794116 (+ 249us) server_negotiation.cc:407] Connection header received
0923 22:20:58.794575 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.794576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.794586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.794709 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.795364 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.795854 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.796760 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.797049 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.797150 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.798114 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:58.798116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:58.798116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:58.798121 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:58.798137 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:58.798142 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:58.798201 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.798277 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.798306 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:20:58.798380 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40747
--builtin_ntp_servers=127.1.129.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 22:20:58.798559 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.793777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:40747 (local address 127.1.129.65:48243)
0923 22:20:58.793982 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.794023 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.794497 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.794781 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.794784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.794793 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:58.795292 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.795296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.795940 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.795941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.796689 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.796691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.797205 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.797206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.797355 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.797945 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:20:58.797946 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:58.797987 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:20:58.798016 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:58.798206 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:58.798210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:58.798212 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.798265 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.798344 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260923 22:20:58.799013 20334 heartbeater.cc:347] Connected to a master server at 127.1.129.126:40747
I20260923 22:20:58.799100 20334 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:58.799284 20334 heartbeater.cc:511] Master 127.1.129.126:40747 requested a full tablet report, sending...
I20260923 22:20:58.799825 20146 ts_manager.cc:194] Registered new tserver with Master: eb408166028041ad998e785739efe291 (127.1.129.65:44905)
I20260923 22:20:58.800452 20146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.65:48243
W20260923 22:20:58.879851 20337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:58.880096 20337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:58.880133 20337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:58.881304 20337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:58.881376 20337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:58.881419 20337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:20:58.882925 20337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:58.883205 20337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:58.883524 20337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:58.888823 20343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.888849 20345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.888861 20342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:58.889146 20337 server_base.cc:1034] running on GCE node
I20260923 22:20:58.889366 20337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:58.889652 20337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:58.890816 20337 hybrid_clock.cc:648] HybridClock initialized: now 1790202058890799 us; error 26 us; skew 500 ppm
I20260923 22:20:58.892331 20337 webserver.cc:460] Webserver started at http://127.1.129.66:33833/ using document root <none> and password file <none>
I20260923 22:20:58.892606 20337 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:58.892656 20337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:58.892819 20337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:58.893968 20337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "2ed06b40d71649d58e57175031382733"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "491cabb312b378630c7c5d40a4ee3606"
server_key_iv: "89f52a316e2d01af0d1806c7e99158e0"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.894343 20337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "2ed06b40d71649d58e57175031382733"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "491cabb312b378630c7c5d40a4ee3606"
server_key_iv: "89f52a316e2d01af0d1806c7e99158e0"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.895669 20337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:20:58.896381 20351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.896546 20337 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:58.896642 20337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "2ed06b40d71649d58e57175031382733"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-dmw7"
server_key: "491cabb312b378630c7c5d40a4ee3606"
server_key_iv: "89f52a316e2d01af0d1806c7e99158e0"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.896723 20337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:58.907058 20337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:58.907346 20337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:58.907496 20337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:58.907713 20337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:58.908030 20337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:58.908083 20337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.908125 20337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:58.908154 20337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.914098 20337 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:34645
I20260923 22:20:58.914150 20464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:34645 every 8 connection(s)
I20260923 22:20:58.914502 20337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:20:58.919243 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.915226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:33791 (local address 127.1.129.126:40747)
0923 22:20:58.915312 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.915314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.915507 (+ 193us) server_negotiation.cc:407] Connection header received
0923 22:20:58.915970 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.915971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.915984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.916038 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.916650 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.917126 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.918045 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.918219 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.918269 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.919040 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:58.919041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:58.919042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:58.919046 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:58.919058 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:58.919062 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:58.919103 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.919165 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.919187 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 22:20:58.919456 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.915197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:40747 (local address 127.1.129.66:33791)
0923 22:20:58.915385 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.915419 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.915886 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.916089 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.916092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.916100 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:58.916581 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.916585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.917201 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.917202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.917973 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.917974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.918288 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.918289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.918393 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.918915 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:20:58.918915 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:58.918943 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:20:58.918965 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:58.919104 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:58.919106 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:58.919108 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:20:58.919154 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 22:20:58.919225 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260923 22:20:58.920012 20465 heartbeater.cc:347] Connected to a master server at 127.1.129.126:40747
I20260923 22:20:58.920113 20465 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:58.920289 20465 heartbeater.cc:511] Master 127.1.129.126:40747 requested a full tablet report, sending...
I20260923 22:20:58.920647 20146 ts_manager.cc:194] Registered new tserver with Master: 2ed06b40d71649d58e57175031382733 (127.1.129.66:34645)
I20260923 22:20:58.921016 20146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.66:33791
I20260923 22:20:58.923832 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 20337
I20260923 22:20:58.923986 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:20:58.924218 1541 external_mini_cluster.cc:1428] Setting key 63368199389952492656776a8ec41c2c
I20260923 22:20:58.925168 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40747
--builtin_ntp_servers=127.1.129.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:20:59.005975 20468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:59.006223 20468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:59.006261 20468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:59.007476 20468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:59.007544 20468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:59.007586 20468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:20:59.008944 20468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:40747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:20:59.009215 20468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:59.009570 20468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:59.014778 20474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.014803 20476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:59.014801 20468 server_base.cc:1034] running on GCE node
W20260923 22:20:59.014783 20473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:59.015215 20468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:59.015462 20468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:59.016618 20468 hybrid_clock.cc:648] HybridClock initialized: now 1790202059016596 us; error 36 us; skew 500 ppm
I20260923 22:20:59.017884 20468 webserver.cc:460] Webserver started at http://127.1.129.67:39269/ using document root <none> and password file <none>
I20260923 22:20:59.018137 20468 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:59.018209 20468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:59.018368 20468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:59.019510 20468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "4ddd6a15f59a4534b429b36afc5657e2"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-dmw7"
server_key: "82963ece1b47f3e38138365d6d5fa7f2"
server_key_iv: "8d4878514bb20180bee02c3c4ab5db0c"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.019883 20468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "4ddd6a15f59a4534b429b36afc5657e2"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-dmw7"
server_key: "82963ece1b47f3e38138365d6d5fa7f2"
server_key_iv: "8d4878514bb20180bee02c3c4ab5db0c"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.021167 20468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:20:59.021939 20482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.022116 20468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:20:59.022214 20468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "4ddd6a15f59a4534b429b36afc5657e2"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-dmw7"
server_key: "82963ece1b47f3e38138365d6d5fa7f2"
server_key_iv: "8d4878514bb20180bee02c3c4ab5db0c"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.022296 20468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:59.030997 20468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:59.031275 20468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:59.031416 20468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:59.031677 20468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:59.031992 20468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:59.032047 20468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.032088 20468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:59.032115 20468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.037916 20468 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:42227
I20260923 22:20:59.037961 20595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:42227 every 8 connection(s)
I20260923 22:20:59.038298 20468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:20:59.040005 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 20468
I20260923 22:20:59.040130 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:20:59.040331 1541 external_mini_cluster.cc:1428] Setting key a8bc14e4316dd9c9ab121c7747758dd8
I20260923 22:20:59.043871 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.039101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:49607 (local address 127.1.129.126:40747)
0923 22:20:59.039166 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.039168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.039407 (+ 239us) server_negotiation.cc:407] Connection header received
0923 22:20:59.039997 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.039998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.040008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.040107 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:59.041030 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.041630 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.042553 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.042817 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.042870 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.043609 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:59.043611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:59.043611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:59.043616 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:59.043630 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:59.043635 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:59.043691 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.043762 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.043791 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:20:59.044095 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.039093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:40747 (local address 127.1.129.67:49607)
0923 22:20:59.039291 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.039325 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.039878 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.040189 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.040194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.040204 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:59.040932 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.040940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.041707 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.041708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.042476 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.042478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.042889 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.042890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.043002 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.043472 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:20:59.043473 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:59.043500 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:20:59.043521 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:59.043694 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:59.043699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:59.043702 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.043768 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.043865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260923 22:20:59.044528 20596 heartbeater.cc:347] Connected to a master server at 127.1.129.126:40747
I20260923 22:20:59.044621 20596 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:59.044806 20596 heartbeater.cc:511] Master 127.1.129.126:40747 requested a full tablet report, sending...
I20260923 22:20:59.045197 20146 ts_manager.cc:194] Registered new tserver with Master: 4ddd6a15f59a4534b429b36afc5657e2 (127.1.129.67:42227)
I20260923 22:20:59.045567 20146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.67:49607
I20260923 22:20:59.052068 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:20:59.056483 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.053367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.1.129.126:40747)
0923 22:20:59.053411 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.053412 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.053566 (+ 154us) server_negotiation.cc:407] Connection header received
0923 22:20:59.053634 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.053636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.053651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.053722 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:59.053903 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.054385 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.055230 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.055420 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.055506 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.056190 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:59.056192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:59.056192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:59.056197 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:59.056210 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:59.056215 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:59.056323 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.056389 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.056399 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:20:59.058257 20146 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:20:59.059932 20146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:20:59.066283 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.062335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:42227 (local address 127.0.0.1:37890)
0923 22:20:59.062541 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.062583 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.062679 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.062962 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.062965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.062966 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.063050 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.063052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.064002 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.064003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.065159 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.065162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.065980 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.065981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.066051 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.066065 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.066150 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.066201 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260923 22:20:59.066286 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.062419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.1.129.67:42227)
0923 22:20:59.062822 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.062826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.062836 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:20:59.062874 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.062876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.062892 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.062962 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.063146 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.063914 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.065255 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.065894 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.065981 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.066073 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.066164 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.066201 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":86,"threads_started":1}
I20260923 22:20:59.066776 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.062578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34645 (local address 127.0.0.1:37778)
0923 22:20:59.062778 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.062788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.062832 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.063179 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.063180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.063182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.063217 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.063218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.064456 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.064457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.065543 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.065546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.066501 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.066502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.066563 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.066567 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.066650 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.066690 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":25,"thread_start_us":59,"threads_started":1}
I20260923 22:20:59.066821 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.062518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.1.129.65:44905)
0923 22:20:59.062720 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.062726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.062778 (+ 52us) server_negotiation.cc:407] Connection header received
0923 22:20:59.062906 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.062908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.062926 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.062997 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.063226 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.064365 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.065469 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.066179 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.066252 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.066492 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.066688 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.066734 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260923 22:20:59.066776 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.062382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:44905 (local address 127.0.0.1:50374)
0923 22:20:59.062710 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.062721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.062809 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.063094 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.063095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.063098 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.063142 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.063144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.064463 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.064464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.065381 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.065383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.066520 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.066521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.066568 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.066571 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.066626 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.066689 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":27,"thread_start_us":79,"threads_started":1}
I20260923 22:20:59.067233 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.062803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.1.129.66:34645)
0923 22:20:59.062984 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.062989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.062998 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:20:59.063032 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.063034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.063048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.063120 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.063288 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.064361 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.065624 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.066315 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.066373 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.066461 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.066948 (+ 487us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.066989 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260923 22:20:59.067227 20530 tablet_service.cc:1467] Processing CreateTablet for tablet 16575cc3c80d4f7291ff5c94f4f87612 (DEFAULT_TABLE table=test-table [id=515b18217d9d4e1aa66950f31e8a5525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:59.067600 20530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16575cc3c80d4f7291ff5c94f4f87612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:59.067643 20268 tablet_service.cc:1467] Processing CreateTablet for tablet 16575cc3c80d4f7291ff5c94f4f87612 (DEFAULT_TABLE table=test-table [id=515b18217d9d4e1aa66950f31e8a5525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:59.067989 20268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16575cc3c80d4f7291ff5c94f4f87612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:59.069943 20610 tablet_bootstrap.cc:492] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Bootstrap starting.
I20260923 22:20:59.070341 20399 tablet_service.cc:1467] Processing CreateTablet for tablet 16575cc3c80d4f7291ff5c94f4f87612 (DEFAULT_TABLE table=test-table [id=515b18217d9d4e1aa66950f31e8a5525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:59.070590 20610 tablet_bootstrap.cc:654] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:59.070689 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16575cc3c80d4f7291ff5c94f4f87612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:59.070748 20611 tablet_bootstrap.cc:492] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291: Bootstrap starting.
I20260923 22:20:59.070956 20610 log.cc:824] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:59.071373 20611 tablet_bootstrap.cc:654] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:59.071648 20611 log.cc:824] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:59.071735 20610 tablet_bootstrap.cc:492] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: No bootstrap required, opened a new log
I20260923 22:20:59.071807 20610 ts_tablet_manager.cc:1397] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:20:59.072290 20611 tablet_bootstrap.cc:492] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291: No bootstrap required, opened a new log
I20260923 22:20:59.072372 20611 ts_tablet_manager.cc:1397] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:20:59.073982 20614 tablet_bootstrap.cc:492] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733: Bootstrap starting.
I20260923 22:20:59.074381 20610 raft_consensus.cc:348] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.074571 20610 raft_consensus.cc:374] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:59.074620 20610 raft_consensus.cc:729] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ddd6a15f59a4534b429b36afc5657e2, State: Initialized, Role: FOLLOWER
I20260923 22:20:59.074689 20614 tablet_bootstrap.cc:654] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:59.074769 20610 consensus_queue.cc:260] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.074713 20611 raft_consensus.cc:348] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.074906 20611 raft_consensus.cc:374] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:59.074967 20611 raft_consensus.cc:729] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb408166028041ad998e785739efe291, State: Initialized, Role: FOLLOWER
I20260923 22:20:59.075003 20614 log.cc:824] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:59.075040 20610 ts_tablet_manager.cc:1428] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260923 22:20:59.075085 20611 consensus_queue.cc:260] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.075166 20596 heartbeater.cc:503] Master 127.1.129.126:40747 was elected leader, sending a full tablet report...
I20260923 22:20:59.075342 20611 ts_tablet_manager.cc:1428] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:20:59.075385 20334 heartbeater.cc:503] Master 127.1.129.126:40747 was elected leader, sending a full tablet report...
I20260923 22:20:59.075824 20614 tablet_bootstrap.cc:492] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733: No bootstrap required, opened a new log
I20260923 22:20:59.075893 20614 ts_tablet_manager.cc:1397] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:20:59.077741 20614 raft_consensus.cc:348] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.077925 20614 raft_consensus.cc:374] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:59.077973 20614 raft_consensus.cc:729] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed06b40d71649d58e57175031382733, State: Initialized, Role: FOLLOWER
I20260923 22:20:59.078088 20614 consensus_queue.cc:260] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.078322 20614 ts_tablet_manager.cc:1428] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 22:20:59.078388 20465 heartbeater.cc:503] Master 127.1.129.126:40747 was elected leader, sending a full tablet report...
W20260923 22:20:59.165076 20466 tablet.cc:2369] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:59.282265 20617 raft_consensus.cc:482] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:59.282397 20617 raft_consensus.cc:504] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.282829 20617 leader_election.cc:288] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ddd6a15f59a4534b429b36afc5657e2 (127.1.129.67:42227), 2ed06b40d71649d58e57175031382733 (127.1.129.66:34645)
I20260923 22:20:59.285759 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.283121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:56525 (local address 127.1.129.67:42227)
0923 22:20:59.283183 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.283185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.283196 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:20:59.283272 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.283273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.283284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.283344 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.283519 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.284046 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.284963 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.285430 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.285507 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.285515 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.285657 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.285692 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:20:59.285761 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.283022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:42227 (local address 127.1.129.65:56525)
0923 22:20:59.283084 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.283097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.283183 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.283362 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.283363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.283364 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.283440 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.283442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.284132 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.284133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.284860 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.284862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.285515 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.285517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.285585 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.285588 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.285652 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.285692 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 22:20:59.285874 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.283263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:46089 (local address 127.1.129.66:34645)
0923 22:20:59.283341 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.283344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.283381 (+ 37us) server_negotiation.cc:407] Connection header received
0923 22:20:59.283476 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.283477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.283492 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.283558 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.283721 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.284194 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.285157 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.285529 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.285611 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.285620 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.285776 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.285812 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:20:59.285884 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.283156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34645 (local address 127.1.129.65:46089)
0923 22:20:59.283311 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.283322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.283397 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.283560 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.283561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.283562 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.283644 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.283645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.284277 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.284278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.285041 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.285043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.285622 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.285623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.285694 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.285697 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.285768 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.285806 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260923 22:20:59.286238 20550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16575cc3c80d4f7291ff5c94f4f87612" candidate_uuid: "eb408166028041ad998e785739efe291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ddd6a15f59a4534b429b36afc5657e2" is_pre_election: true
I20260923 22:20:59.286237 20419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16575cc3c80d4f7291ff5c94f4f87612" candidate_uuid: "eb408166028041ad998e785739efe291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed06b40d71649d58e57175031382733" is_pre_election: true
I20260923 22:20:59.286387 20419 raft_consensus.cc:2454] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb408166028041ad998e785739efe291 in term 0.
I20260923 22:20:59.286387 20550 raft_consensus.cc:2454] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb408166028041ad998e785739efe291 in term 0.
I20260923 22:20:59.286623 20223 leader_election.cc:302] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ddd6a15f59a4534b429b36afc5657e2, eb408166028041ad998e785739efe291; no voters:
I20260923 22:20:59.286769 20617 raft_consensus.cc:2781] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:59.286836 20617 raft_consensus.cc:482] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:59.286870 20617 raft_consensus.cc:3037] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:59.287616 20617 raft_consensus.cc:504] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.287793 20617 leader_election.cc:288] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [CANDIDATE]: Term 1 election: Requested vote from peers 4ddd6a15f59a4534b429b36afc5657e2 (127.1.129.67:42227), 2ed06b40d71649d58e57175031382733 (127.1.129.66:34645)
I20260923 22:20:59.287940 20550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16575cc3c80d4f7291ff5c94f4f87612" candidate_uuid: "eb408166028041ad998e785739efe291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ddd6a15f59a4534b429b36afc5657e2"
I20260923 22:20:59.287961 20419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16575cc3c80d4f7291ff5c94f4f87612" candidate_uuid: "eb408166028041ad998e785739efe291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed06b40d71649d58e57175031382733"
I20260923 22:20:59.288022 20550 raft_consensus.cc:3037] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:59.288035 20419 raft_consensus.cc:3037] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:59.288766 20550 raft_consensus.cc:2454] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb408166028041ad998e785739efe291 in term 1.
W20260923 22:20:59.288811 20597 tablet.cc:2369] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:59.288872 20419 raft_consensus.cc:2454] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb408166028041ad998e785739efe291 in term 1.
I20260923 22:20:59.288931 20223 leader_election.cc:302] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ddd6a15f59a4534b429b36afc5657e2, eb408166028041ad998e785739efe291; no voters:
I20260923 22:20:59.289009 20617 raft_consensus.cc:2781] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:59.289157 20617 raft_consensus.cc:686] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [term 1 LEADER]: Becoming Leader. State: Replica: eb408166028041ad998e785739efe291, State: Running, Role: LEADER
I20260923 22:20:59.289250 20617 consensus_queue.cc:237] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } }
I20260923 22:20:59.290042 20146 catalog_manager.cc:5495] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb408166028041ad998e785739efe291 (127.1.129.65). New cstate: current_term: 1 leader_uuid: "eb408166028041ad998e785739efe291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb408166028041ad998e785739efe291" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 44905 } health_report { overall_health: HEALTHY } } }
W20260923 22:20:59.293730 20335 tablet.cc:2369] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:59.348810 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.346548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.1.129.65:44905)
0923 22:20:59.346596 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.346598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.346606 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:59.346697 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.346698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.346724 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.346798 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:59.346918 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.347387 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.348089 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.348272 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.348323 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.348451 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:59.348538 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:59.348608 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.348661 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.348670 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:20:59.350677 20419 raft_consensus.cc:1261] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 1 FOLLOWER]: Refusing update from remote peer eb408166028041ad998e785739efe291: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:59.350745 20550 raft_consensus.cc:1261] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 1 FOLLOWER]: Refusing update from remote peer eb408166028041ad998e785739efe291: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:59.351035 20617 consensus_queue.cc:1035] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ed06b40d71649d58e57175031382733" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:59.351178 20620 consensus_queue.cc:1035] T 16575cc3c80d4f7291ff5c94f4f87612 P eb408166028041ad998e785739efe291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ddd6a15f59a4534b429b36afc5657e2" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:59.353597 20623 mvcc.cc:205] Tried to move back new op lower bound from 7332667635097546752 to 7332667634849488896. Current Snapshot: MvccSnapshot[applied={T|T < 7332667635097546752}]
I20260923 22:20:59.353688 20624 mvcc.cc:205] Tried to move back new op lower bound from 7332667635097546752 to 7332667634849488896. Current Snapshot: MvccSnapshot[applied={T|T < 7332667635097546752}]
I20260923 22:20:59.353708 20625 mvcc.cc:205] Tried to move back new op lower bound from 7332667635097546752 to 7332667634849488896. Current Snapshot: MvccSnapshot[applied={T|T < 7332667635097546752}]
I20260923 22:20:59.356040 20144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:59.356137 20144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:59.356977 20144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58ba5449ff447b68c76770c6284ae19: Sending DeleteTablet for 3 replicas of tablet 16575cc3c80d4f7291ff5c94f4f87612
I20260923 22:20:59.357285 20530 tablet_service.cc:1514] Processing DeleteTablet for tablet 16575cc3c80d4f7291ff5c94f4f87612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:59 UTC) from {username='slave'} at 127.0.0.1:37890
I20260923 22:20:59.357309 20399 tablet_service.cc:1514] Processing DeleteTablet for tablet 16575cc3c80d4f7291ff5c94f4f87612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:59 UTC) from {username='slave'} at 127.0.0.1:37778
I20260923 22:20:59.357537 20268 tablet_service.cc:1514] Processing DeleteTablet for tablet 16575cc3c80d4f7291ff5c94f4f87612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:59 UTC) from {username='slave'} at 127.0.0.1:50374
I20260923 22:20:59.357599 20633 tablet_replica.cc:330] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: stopping tablet replica
I20260923 22:20:59.357712 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 20206
I20260923 22:20:59.359391 20633 raft_consensus.cc:2229] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:59.359529 20633 raft_consensus.cc:2258] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:59.359910 20633 ts_tablet_manager.cc:1905] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:59.361466 20633 ts_tablet_manager.cc:1918] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:59.361529 20633 log.cc:1196] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/16575cc3c80d4f7291ff5c94f4f87612
I20260923 22:20:59.361783 20633 ts_tablet_manager.cc:1939] T 16575cc3c80d4f7291ff5c94f4f87612 P 4ddd6a15f59a4534b429b36afc5657e2: Deleting consensus metadata
I20260923 22:20:59.362193 20132 catalog_manager.cc:4841] TS 4ddd6a15f59a4534b429b36afc5657e2 (127.1.129.67:42227): tablet 16575cc3c80d4f7291ff5c94f4f87612 (table test-table [id=515b18217d9d4e1aa66950f31e8a5525]) successfully deleted
W20260923 22:20:59.362596 20130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:44905 (error 108)
I20260923 22:20:59.357760 20634 tablet_replica.cc:330] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733: stopping tablet replica
I20260923 22:20:59.363133 20634 raft_consensus.cc:2229] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:59.363194 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.363061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:44905 (local address 127.0.0.1:50392)
0923 22:20:59.363109 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.363161 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:44905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 22:20:59.363178 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 20337
I20260923 22:20:59.363281 20634 raft_consensus.cc:2258] T 16575cc3c80d4f7291ff5c94f4f87612 P 2ed06b40d71649d58e57175031382733 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 22:20:59.363293 20130 catalog_manager.cc:4584] TS eb408166028041ad998e785739efe291 (127.1.129.65:44905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16575cc3c80d4f7291ff5c94f4f87612: Network error: Client connection negotiation failed: client connection to 127.1.129.65:44905: connect: Connection refused (error 111)
I20260923 22:20:59.367399 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 20468
I20260923 22:20:59.367456 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.367344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:34645 (local address 127.0.0.1:37792)
0923 22:20:59.367400 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.367432 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.66:34645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 22:20:59.372114 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 20115
2026-09-23T22:20:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (908 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:20:59 dist-test-slave-dmw7 krb5kdc[20639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:20:59 dist-test-slave-dmw7 krb5kdc[20639](info): set up 2 sockets
Sep 23 22:20:59 dist-test-slave-dmw7 krb5kdc[20639](info): commencing operation
krb5kdc: starting...
W20260923 22:21:00.414530 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:00 dist-test-slave-dmw7 krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:00.539583 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:32859
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:32859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:00.623567 20652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:00.623786 20652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:00.623809 20652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:00.625034 20652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:00.625077 20652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:00.625092 20652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:00.625104 20652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:00.625118 20652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:00.626703 20652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:32859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:32859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:00.626910 20652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:00.627106 20652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:00.632048 20658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:00.632077 20660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:00.632048 20657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:00.632407 20652 server_base.cc:1034] running on GCE node
I20260923 22:21:00.632735 20652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:00.633054 20652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:00.634231 20652 hybrid_clock.cc:648] HybridClock initialized: now 1790202060634211 us; error 30 us; skew 500 ppm
Sep 23 22:21:00 dist-test-slave-dmw7 krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:00.637023 20652 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:00.637526 20652 webserver.cc:460] Webserver started at http://127.1.129.126:42253/ using document root <none> and password file <none>
I20260923 22:21:00.637778 20652 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:00.637854 20652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:00.638027 20652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:00.639276 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "dbcf8c6ec067404fa7bbccfeca040439"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-dmw7"
server_key: "09ce8e4b3cce05a9f206fa379438c93d"
server_key_iv: "037bbb2927e6ea3898f835a524401d50"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.639683 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "dbcf8c6ec067404fa7bbccfeca040439"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-dmw7"
server_key: "09ce8e4b3cce05a9f206fa379438c93d"
server_key_iv: "037bbb2927e6ea3898f835a524401d50"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.641203 20652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:00.642014 20667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.642257 20652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:00.642378 20652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "dbcf8c6ec067404fa7bbccfeca040439"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-dmw7"
server_key: "09ce8e4b3cce05a9f206fa379438c93d"
server_key_iv: "037bbb2927e6ea3898f835a524401d50"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.642458 20652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:00.663530 20652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:00.665545 20652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:00.665714 20652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:00.669729 20652 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:32859
I20260923 22:21:00.669766 20719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:32859 every 8 connection(s)
I20260923 22:21:00.670146 20652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:00.671100 20720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:00.673348 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439: Bootstrap starting.
I20260923 22:21:00.673868 20720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:00.674103 20720 log.cc:824] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:00.674791 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439: No bootstrap required, opened a new log
I20260923 22:21:00.675472 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 20652
I20260923 22:21:00.675586 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:00.675791 1541 external_mini_cluster.cc:1428] Setting key 23e4a46116e42f83d82cd01dbe12e317
I20260923 22:21:00.677055 20720 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf8c6ec067404fa7bbccfeca040439" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 32859 } }
I20260923 22:21:00.677191 20720 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:00.677211 20720 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcf8c6ec067404fa7bbccfeca040439, State: Initialized, Role: FOLLOWER
I20260923 22:21:00.677345 20720 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf8c6ec067404fa7bbccfeca040439" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 32859 } }
I20260923 22:21:00.677417 20720 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:00.677439 20720 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:00.677505 20720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:00.678251 20720 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf8c6ec067404fa7bbccfeca040439" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 32859 } }
I20260923 22:21:00.678370 20720 leader_election.cc:302] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbcf8c6ec067404fa7bbccfeca040439; no voters:
I20260923 22:21:00.678576 20720 leader_election.cc:288] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:00.678610 20725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:00.678742 20725 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [term 1 LEADER]: Becoming Leader. State: Replica: dbcf8c6ec067404fa7bbccfeca040439, State: Running, Role: LEADER
I20260923 22:21:00.678833 20725 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf8c6ec067404fa7bbccfeca040439" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 32859 } }
I20260923 22:21:00.678864 20720 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:00.679200 20726 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbcf8c6ec067404fa7bbccfeca040439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf8c6ec067404fa7bbccfeca040439" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 32859 } } }
I20260923 22:21:00.679364 20726 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:00.679888 20727 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcf8c6ec067404fa7bbccfeca040439. Latest consensus state: current_term: 1 leader_uuid: "dbcf8c6ec067404fa7bbccfeca040439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf8c6ec067404fa7bbccfeca040439" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 32859 } } }
I20260923 22:21:00.679971 20727 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:00.680208 20733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:00.680686 20733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:00.682215 20733 catalog_manager.cc:1347] Generated new cluster ID: 1c6ba5bbb4be40119f7134ea54b8c9ca
I20260923 22:21:00.682271 20733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:21:00 dist-test-slave-dmw7 krb5kdc[20639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:00.687584 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.676325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.1.129.126:32859)
0923 22:21:00.676491 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.676500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.676534 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:00.677229 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.677239 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.677332 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.677504 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:00.678427 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.679861 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.680894 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.681457 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.681565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.684324 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:00.684328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:00.684329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:00.684341 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:00.685976 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:00.686356 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:00.686359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:00.686359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:00.686457 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:00.686708 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:00.686709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:00.686709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:00.686962 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:00.687108 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.687223 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.687368 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260923 22:21:00.693674 20733 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:00.694262 20733 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:00.702301 20733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439: Generated new TSK 0
I20260923 22:21:00.702520 20733 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.129.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
Entry for principal kudu/127.1.129.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab.
I20260923 22:21:00.739909 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:32859
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:00.857544 20746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:00.857822 20746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:00.857859 20746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:00.859400 20746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:00.859452 20746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:00.859488 20746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:21:00.861373 20746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.65.keytab
--principal=kudu/127.1.129.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:32859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:00.861565 20746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:00.861732 20746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:21:00.866698 20746 server_base.cc:1034] running on GCE node
W20260923 22:21:00.866680 20752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:00.866675 20751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:00.866680 20754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:00.867110 20746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:00.867367 20746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:00.868532 20746 hybrid_clock.cc:648] HybridClock initialized: now 1790202060868508 us; error 33 us; skew 500 ppm
Sep 23 22:21:00 dist-test-slave-dmw7 krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:00.871161 20746 init.cc:377] Logged in from keytab as kudu/127.1.129.65@KRBTEST.COM (short username kudu)
I20260923 22:21:00.871640 20746 webserver.cc:460] Webserver started at http://127.1.129.65:40305/ using document root <none> and password file <none>
I20260923 22:21:00.871881 20746 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:00.871946 20746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:00.872087 20746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:00.873112 20746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "06d8237bcd3548d0a51b2832e777e2bb"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-dmw7"
server_key: "bd28bb8449302b07407a786b45f4a5f8"
server_key_iv: "7f261e284c9c01619882f77b7580f2aa"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.873466 20746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "06d8237bcd3548d0a51b2832e777e2bb"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-dmw7"
server_key: "bd28bb8449302b07407a786b45f4a5f8"
server_key_iv: "7f261e284c9c01619882f77b7580f2aa"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.874759 20746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:00.875468 20761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.875635 20746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:00.875726 20746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "06d8237bcd3548d0a51b2832e777e2bb"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-dmw7"
server_key: "bd28bb8449302b07407a786b45f4a5f8"
server_key_iv: "7f261e284c9c01619882f77b7580f2aa"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.875805 20746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:00.892762 20746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:00.895076 20746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:00.895236 20746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:00.895459 20746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:00.895783 20746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:00.895831 20746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.895864 20746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:00.895893 20746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.901476 20746 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:33639
I20260923 22:21:00.901525 20874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:33639 every 8 connection(s)
I20260923 22:21:00.901813 20746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:21:00.905655 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 20746
I20260923 22:21:00.905771 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:21:00.905974 1541 external_mini_cluster.cc:1428] Setting key 970291ae631a012d6a5052416fde8fd2
Sep 23 22:21:00 dist-test-slave-dmw7 krb5kdc[20639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.65@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:00.911600 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.902388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:32859 (local address 127.1.129.65:52909)
0923 22:21:00.902547 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.902589 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:00.903094 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:00.903321 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:00.903324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:00.903861 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:00.904045 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.904049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.904702 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.904703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.905467 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.905468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.906049 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.906051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.906201 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.906832 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:00.906840 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:00.908380 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:00.910255 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:00.910261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:00.910263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:00.910483 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:00.910711 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:00.910712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:00.910713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:00.910811 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:00.911103 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:00.911109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:00.911224 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 22:21:00.911315 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 22:21:00.911438 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260923 22:21:00.911608 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.902432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:52909 (local address 127.1.129.126:32859)
0923 22:21:00.902493 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.902495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.902688 (+ 193us) server_negotiation.cc:407] Connection header received
0923 22:21:00.903178 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.903179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.903189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.903246 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:00.904117 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.904629 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.905552 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.905893 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.905970 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.908478 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:00.908480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:00.908480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:00.908487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:00.910165 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:00.910541 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:00.910543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:00.910543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:00.910608 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:00.910870 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:00.910871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:00.910871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:00.911014 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:00.911088 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.911363 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.911464 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:21:00.912096 20875 heartbeater.cc:347] Connected to a master server at 127.1.129.126:32859
I20260923 22:21:00.912204 20875 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:00.912411 20875 heartbeater.cc:511] Master 127.1.129.126:32859 requested a full tablet report, sending...
I20260923 22:21:00.912919 20684 ts_manager.cc:194] Registered new tserver with Master: 06d8237bcd3548d0a51b2832e777e2bb (127.1.129.65:33639)
I20260923 22:21:00.913527 20684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.65@KRBTEST.COM'} at 127.1.129.65:52909
WARNING: no policy specified for kudu/127.1.129.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
Entry for principal kudu/127.1.129.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab.
I20260923 22:21:00.931838 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:32859
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:01.011725 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.011915 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.011940 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.012986 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.013018 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.013044 20880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:21:01.014389 20880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.66.keytab
--principal=kudu/127.1.129.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:32859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:01.014564 20880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.014786 20880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:01.019642 20886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.019675 20880 server_base.cc:1034] running on GCE node
W20260923 22:21:01.019695 20888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.019639 20885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.020092 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.020327 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.021483 20880 hybrid_clock.cc:648] HybridClock initialized: now 1790202061021473 us; error 35 us; skew 500 ppm
Sep 23 22:21:01 dist-test-slave-dmw7 krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:01.023895 20880 init.cc:377] Logged in from keytab as kudu/127.1.129.66@KRBTEST.COM (short username kudu)
I20260923 22:21:01.024308 20880 webserver.cc:460] Webserver started at http://127.1.129.66:40347/ using document root <none> and password file <none>
I20260923 22:21:01.024541 20880 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.024606 20880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.024747 20880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.025717 20880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "d9e666c8c6f140718b61ebac9a53ea28"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "ba9117322b1a46f8e1ea19e14b9bdc91"
server_key_iv: "4b42aeb0fbb7b83b9ef2e230df88fca7"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.026058 20880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "d9e666c8c6f140718b61ebac9a53ea28"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "ba9117322b1a46f8e1ea19e14b9bdc91"
server_key_iv: "4b42aeb0fbb7b83b9ef2e230df88fca7"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.027434 20880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:01.028158 20895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.028313 20880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:01.028406 20880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "d9e666c8c6f140718b61ebac9a53ea28"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "ba9117322b1a46f8e1ea19e14b9bdc91"
server_key_iv: "4b42aeb0fbb7b83b9ef2e230df88fca7"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.028486 20880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.042994 20880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.045305 20880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.045415 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.045595 20880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:01.045886 20880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:01.045918 20880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.045959 20880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:01.045986 20880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.051672 20880 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:40931
I20260923 22:21:01.051713 21008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:40931 every 8 connection(s)
I20260923 22:21:01.052045 20880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:21:01.056334 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 20880
I20260923 22:21:01.056464 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:21:01.056685 1541 external_mini_cluster.cc:1428] Setting key 90bb3d1801306cd2cbc033cb61b1f6bb
Sep 23 22:21:01 dist-test-slave-dmw7 krb5kdc[20639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.66@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:01.061513 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.052751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:58231 (local address 127.1.129.126:32859)
0923 22:21:01.052808 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.052810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.052992 (+ 182us) server_negotiation.cc:407] Connection header received
0923 22:21:01.053478 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.053479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.053488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.053532 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.054446 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.054953 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.055895 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.056074 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.056128 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.058614 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.058615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.058616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:01.058624 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.060153 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.060591 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.060593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.060594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.060652 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.060894 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.060895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.060895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.061004 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.061079 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.061338 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.061422 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 22:21:01.061720 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.052713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:32859 (local address 127.1.129.66:58231)
0923 22:21:01.052869 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.052912 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.053378 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.053619 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.053622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.054175 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:01.054370 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.054375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.055038 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.055039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.055824 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.055826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.056142 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.056143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.056237 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.056783 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:01.056792 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:01.058524 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:01.060252 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:01.060260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:01.060262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:01.060489 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:01.060724 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:01.060725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:01.060726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:01.060830 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:01.061069 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:01.061075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:01.061214 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.061317 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.061474 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260923 22:21:01.062199 21009 heartbeater.cc:347] Connected to a master server at 127.1.129.126:32859
I20260923 22:21:01.062309 21009 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:01.062503 21009 heartbeater.cc:511] Master 127.1.129.126:32859 requested a full tablet report, sending...
I20260923 22:21:01.062914 20684 ts_manager.cc:194] Registered new tserver with Master: d9e666c8c6f140718b61ebac9a53ea28 (127.1.129.66:40931)
I20260923 22:21:01.063441 20684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.66@KRBTEST.COM'} at 127.1.129.66:58231
WARNING: no policy specified for kudu/127.1.129.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
Entry for principal kudu/127.1.129.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab.
I20260923 22:21:01.080557 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:32859
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:01.159157 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.159345 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.159368 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.160452 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.160485 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.160508 21014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:21:01.161782 21014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.67.keytab
--principal=kudu/127.1.129.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:32859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:01.161939 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.162093 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:21:01.166896 21014 server_base.cc:1034] running on GCE node
W20260923 22:21:01.166872 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.166872 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.166913 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.167310 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.167549 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.168701 21014 hybrid_clock.cc:648] HybridClock initialized: now 1790202061168683 us; error 32 us; skew 500 ppm
Sep 23 22:21:01 dist-test-slave-dmw7 krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:01.171128 21014 init.cc:377] Logged in from keytab as kudu/127.1.129.67@KRBTEST.COM (short username kudu)
I20260923 22:21:01.171535 21014 webserver.cc:460] Webserver started at http://127.1.129.67:41589/ using document root <none> and password file <none>
I20260923 22:21:01.171741 21014 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.171803 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.171940 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.172894 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "f56ce126d7cb4130a4df5a8095dbff59"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "bb13495c1040913b4cbd160180891bc2"
server_key_iv: "01dca8129c1afc6aae52ddbc483ffe75"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.173235 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "f56ce126d7cb4130a4df5a8095dbff59"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "bb13495c1040913b4cbd160180891bc2"
server_key_iv: "01dca8129c1afc6aae52ddbc483ffe75"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.174477 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:21:01.175216 21029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.175381 21014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.175469 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "f56ce126d7cb4130a4df5a8095dbff59"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "bb13495c1040913b4cbd160180891bc2"
server_key_iv: "01dca8129c1afc6aae52ddbc483ffe75"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.175554 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.185297 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.187160 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.187317 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.187543 21014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:01.187827 21014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:01.187876 21014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.187918 21014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:01.187947 21014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.193708 21014 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:42087
I20260923 22:21:01.193751 21142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:42087 every 8 connection(s)
I20260923 22:21:01.194042 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:21:01.194739 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 21014
I20260923 22:21:01.194829 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:21:01.195003 1541 external_mini_cluster.cc:1428] Setting key 913963763a6abb1166973c2baaa331e8
Sep 23 22:21:01 dist-test-slave-dmw7 krb5kdc[20639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.67@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:01.203236 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.194713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:39073 (local address 127.1.129.126:32859)
0923 22:21:01.194771 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.194774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.195351 (+ 577us) server_negotiation.cc:407] Connection header received
0923 22:21:01.195976 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.195977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.195988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.196033 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.196935 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.197426 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.198423 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.198621 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.198747 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.200685 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.200687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.200687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:01.200695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.201816 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.202246 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.202248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.202248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.202309 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.202549 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.202551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.202551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.202722 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.202797 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.203068 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.203146 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 22:21:01.203425 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.194715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:32859 (local address 127.1.129.67:39073)
0923 22:21:01.195200 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.195253 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.195873 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.196115 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.196120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.196667 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:01.196848 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.196852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.197513 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.197514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.198323 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.198325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.198755 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.198757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.198888 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.199322 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:01.199329 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:01.200598 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:01.201895 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:01.201902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:01.201905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:01.202128 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:01.202372 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:01.202373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:01.202373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:01.202473 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:01.202781 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:01.202788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:01.202924 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.203043 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.203193 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":79,"threads_started":1}
I20260923 22:21:01.203917 21143 heartbeater.cc:347] Connected to a master server at 127.1.129.126:32859
I20260923 22:21:01.204120 21143 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:01.204308 21143 heartbeater.cc:511] Master 127.1.129.126:32859 requested a full tablet report, sending...
I20260923 22:21:01.204674 20684 ts_manager.cc:194] Registered new tserver with Master: f56ce126d7cb4130a4df5a8095dbff59 (127.1.129.67:42087)
I20260923 22:21:01.204994 20684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.129.67@KRBTEST.COM'} at 127.1.129.67:39073
I20260923 22:21:01.206743 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:21:01.213285 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.208017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.1.129.126:32859)
0923 22:21:01.208106 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.208108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.208161 (+ 53us) server_negotiation.cc:407] Connection header received
0923 22:21:01.208210 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.208210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.208218 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.208274 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.208744 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.209190 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.209847 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.210018 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.210067 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.211285 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.211286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.211286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:01.211292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.212411 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.212709 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.212710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.212710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.212755 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.212902 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.212902 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.212902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.213030 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.213102 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.213179 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.213228 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 22:21:01.214710 20684 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:21:01.216212 20684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:21:01.222626 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.218746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:42087 (local address 127.0.0.1:47662)
0923 22:21:01.218921 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.218931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.218977 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.219262 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.219263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.219265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:01.219325 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.219326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.220608 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.220609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.221428 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.221431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.222334 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.222335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.222419 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.222432 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.222497 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.222556 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":59,"thread_start_us":64,"threads_started":1}
I20260923 22:21:01.222754 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.218733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.1.129.65:33639)
0923 22:21:01.219080 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.219084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.219094 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:01.219129 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.219131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.219146 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.219377 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:01.219451 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.220126 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.221656 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.222334 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.222426 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.222478 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.222574 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.222667 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":80,"threads_started":1}
I20260923 22:21:01.222813 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.218612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.1.129.66:40931)
0923 22:21:01.218831 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.218836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.218846 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:01.218888 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.218891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.218908 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.218986 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:01.219496 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.220559 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.221778 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.222265 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.222346 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.222410 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.222685 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.222730 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
I20260923 22:21:01.222992 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.218913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.1.129.67:42087)
0923 22:21:01.219098 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.219103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.219112 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:21:01.219151 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.219153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.219167 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.219267 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:01.219386 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.220490 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.221531 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.222247 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.222322 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.222379 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.222868 (+ 489us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.222915 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260923 22:21:01.223147 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.218526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:33639 (local address 127.0.0.1:57394)
0923 22:21:01.218903 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.218913 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.218964 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.219241 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.219241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.219243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:01.219301 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.219303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.220221 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.220222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.221586 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.221588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.222433 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.222434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.222499 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.222503 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.222560 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.222618 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260923 22:21:01.223479 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.218515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:40931 (local address 127.0.0.1:49272)
0923 22:21:01.218667 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.218707 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.218765 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.219097 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.219101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.219103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:01.219411 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.219413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.220657 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.220658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.221703 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.221704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.222442 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.222442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.222517 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.222520 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.222563 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.222609 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260923 22:21:01.223732 21077 tablet_service.cc:1467] Processing CreateTablet for tablet f2b1932805bd4f7eac69d2fdf24b65be (DEFAULT_TABLE table=test-table [id=0934d5ad33bf4511a7cad27e9110cae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:01.223809 20809 tablet_service.cc:1467] Processing CreateTablet for tablet f2b1932805bd4f7eac69d2fdf24b65be (DEFAULT_TABLE table=test-table [id=0934d5ad33bf4511a7cad27e9110cae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:01.224097 21077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b1932805bd4f7eac69d2fdf24b65be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:01.224109 20809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b1932805bd4f7eac69d2fdf24b65be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:01.224102 20943 tablet_service.cc:1467] Processing CreateTablet for tablet f2b1932805bd4f7eac69d2fdf24b65be (DEFAULT_TABLE table=test-table [id=0934d5ad33bf4511a7cad27e9110cae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:01.224375 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b1932805bd4f7eac69d2fdf24b65be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:01.226846 21160 tablet_bootstrap.cc:492] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb: Bootstrap starting.
I20260923 22:21:01.227417 21161 tablet_bootstrap.cc:492] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: Bootstrap starting.
I20260923 22:21:01.227618 21162 tablet_bootstrap.cc:492] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28: Bootstrap starting.
I20260923 22:21:01.227771 21160 tablet_bootstrap.cc:654] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:01.228098 21161 tablet_bootstrap.cc:654] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:01.228188 21162 tablet_bootstrap.cc:654] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:01.228364 21160 log.cc:824] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:01.228691 21161 log.cc:824] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:01.228700 21162 log.cc:824] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:01.229431 21162 tablet_bootstrap.cc:492] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28: No bootstrap required, opened a new log
I20260923 22:21:01.229504 21162 ts_tablet_manager.cc:1397] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:01.230978 21160 tablet_bootstrap.cc:492] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb: No bootstrap required, opened a new log
I20260923 22:21:01.231057 21160 ts_tablet_manager.cc:1397] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 22:21:01.230940 21162 raft_consensus.cc:348] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.231107 21162 raft_consensus.cc:374] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:01.231153 21162 raft_consensus.cc:729] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e666c8c6f140718b61ebac9a53ea28, State: Initialized, Role: FOLLOWER
I20260923 22:21:01.231262 21162 consensus_queue.cc:260] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.231539 21009 heartbeater.cc:503] Master 127.1.129.126:32859 was elected leader, sending a full tablet report...
I20260923 22:21:01.231544 21162 ts_tablet_manager.cc:1428] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:21:01.232812 21160 raft_consensus.cc:348] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.232973 21160 raft_consensus.cc:374] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:01.233018 21160 raft_consensus.cc:729] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d8237bcd3548d0a51b2832e777e2bb, State: Initialized, Role: FOLLOWER
I20260923 22:21:01.233129 21160 consensus_queue.cc:260] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.233420 21160 ts_tablet_manager.cc:1428] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:21:01.233429 20875 heartbeater.cc:503] Master 127.1.129.126:32859 was elected leader, sending a full tablet report...
I20260923 22:21:01.233804 21161 tablet_bootstrap.cc:492] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: No bootstrap required, opened a new log
I20260923 22:21:01.233851 21161 ts_tablet_manager.cc:1397] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260923 22:21:01.235308 21161 raft_consensus.cc:348] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.235426 21161 raft_consensus.cc:374] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:01.235447 21161 raft_consensus.cc:729] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56ce126d7cb4130a4df5a8095dbff59, State: Initialized, Role: FOLLOWER
I20260923 22:21:01.235575 21161 consensus_queue.cc:260] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.235828 21161 ts_tablet_manager.cc:1428] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:21:01.235834 21143 heartbeater.cc:503] Master 127.1.129.126:32859 was elected leader, sending a full tablet report...
I20260923 22:21:01.239864 21168 raft_consensus.cc:482] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:21:01.239933 21168 raft_consensus.cc:504] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.240162 21168 leader_election.cc:288] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9e666c8c6f140718b61ebac9a53ea28 (127.1.129.66:40931), 06d8237bcd3548d0a51b2832e777e2bb (127.1.129.65:33639)
I20260923 22:21:01.242992 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.240426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:53633 (local address 127.1.129.66:40931)
0923 22:21:01.240464 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.240466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.240475 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:21:01.240512 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.240513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.240524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.240580 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:01.240738 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.241230 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.242120 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.242678 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.242742 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.242750 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.242895 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.242929 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:21:01.243005 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.240313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:40931 (local address 127.1.129.67:53633)
0923 22:21:01.240365 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.240377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.240416 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.240592 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.240593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.240595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:01.240671 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.240673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.241301 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.241302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.242043 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.242045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.242738 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.242739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.242815 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.242819 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.242888 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.242928 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":21}
I20260923 22:21:01.243111 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.240371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:34023 (local address 127.1.129.65:33639)
0923 22:21:01.240417 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.240420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.240580 (+ 160us) server_negotiation.cc:407] Connection header received
0923 22:21:01.240610 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.240611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.240620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.240677 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:01.240822 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.241249 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.242249 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.242773 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.242860 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.242868 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.243017 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.243052 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:21:01.243129 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.240323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:33639 (local address 127.1.129.67:34023)
0923 22:21:01.240496 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.240506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.240568 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.240684 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.240685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.240686 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:01.240761 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.240763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.241344 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.241345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.242064 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.242065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.242863 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.242864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.242932 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.242936 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.243010 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.243047 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260923 22:21:01.243427 20963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b1932805bd4f7eac69d2fdf24b65be" candidate_uuid: "f56ce126d7cb4130a4df5a8095dbff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e666c8c6f140718b61ebac9a53ea28" is_pre_election: true
I20260923 22:21:01.243470 20829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b1932805bd4f7eac69d2fdf24b65be" candidate_uuid: "f56ce126d7cb4130a4df5a8095dbff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d8237bcd3548d0a51b2832e777e2bb" is_pre_election: true
I20260923 22:21:01.243570 20829 raft_consensus.cc:2454] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56ce126d7cb4130a4df5a8095dbff59 in term 0.
I20260923 22:21:01.243570 20963 raft_consensus.cc:2454] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f56ce126d7cb4130a4df5a8095dbff59 in term 0.
I20260923 22:21:01.243765 21033 leader_election.cc:302] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d8237bcd3548d0a51b2832e777e2bb, f56ce126d7cb4130a4df5a8095dbff59; no voters:
I20260923 22:21:01.243882 21168 raft_consensus.cc:2781] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:21:01.243958 21168 raft_consensus.cc:482] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:21:01.243997 21168 raft_consensus.cc:3037] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:01.244642 21168 raft_consensus.cc:504] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.244791 21168 leader_election.cc:288] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [CANDIDATE]: Term 1 election: Requested vote from peers d9e666c8c6f140718b61ebac9a53ea28 (127.1.129.66:40931), 06d8237bcd3548d0a51b2832e777e2bb (127.1.129.65:33639)
I20260923 22:21:01.244946 20829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b1932805bd4f7eac69d2fdf24b65be" candidate_uuid: "f56ce126d7cb4130a4df5a8095dbff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d8237bcd3548d0a51b2832e777e2bb"
I20260923 22:21:01.245005 20963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b1932805bd4f7eac69d2fdf24b65be" candidate_uuid: "f56ce126d7cb4130a4df5a8095dbff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e666c8c6f140718b61ebac9a53ea28"
I20260923 22:21:01.245041 20829 raft_consensus.cc:3037] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:01.245093 20963 raft_consensus.cc:3037] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:01.245800 20963 raft_consensus.cc:2454] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56ce126d7cb4130a4df5a8095dbff59 in term 1.
I20260923 22:21:01.245836 20829 raft_consensus.cc:2454] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f56ce126d7cb4130a4df5a8095dbff59 in term 1.
I20260923 22:21:01.245972 21032 leader_election.cc:302] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9e666c8c6f140718b61ebac9a53ea28, f56ce126d7cb4130a4df5a8095dbff59; no voters:
I20260923 22:21:01.246095 21168 raft_consensus.cc:2781] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:01.246248 21168 raft_consensus.cc:686] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 1 LEADER]: Becoming Leader. State: Replica: f56ce126d7cb4130a4df5a8095dbff59, State: Running, Role: LEADER
I20260923 22:21:01.246336 21168 consensus_queue.cc:237] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } }
I20260923 22:21:01.247129 20684 catalog_manager.cc:5495] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to f56ce126d7cb4130a4df5a8095dbff59 (127.1.129.67). New cstate: current_term: 1 leader_uuid: "f56ce126d7cb4130a4df5a8095dbff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f56ce126d7cb4130a4df5a8095dbff59" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 42087 } health_report { overall_health: HEALTHY } } }
I20260923 22:21:01.252391 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.250098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.1.129.67:42087)
0923 22:21:01.250133 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.250135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.250177 (+ 42us) server_negotiation.cc:407] Connection header received
0923 22:21:01.250248 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.250249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.250261 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.250330 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:01.250452 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.250934 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.251657 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.251832 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.251891 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.252070 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:01.252153 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:01.252227 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.252314 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.252331 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:01.253894 20963 raft_consensus.cc:1261] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 1 FOLLOWER]: Refusing update from remote peer f56ce126d7cb4130a4df5a8095dbff59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:01.253893 20829 raft_consensus.cc:1261] T f2b1932805bd4f7eac69d2fdf24b65be P 06d8237bcd3548d0a51b2832e777e2bb [term 1 FOLLOWER]: Refusing update from remote peer f56ce126d7cb4130a4df5a8095dbff59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:01.254251 21170 consensus_queue.cc:1035] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9e666c8c6f140718b61ebac9a53ea28" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:01.254355 21168 consensus_queue.cc:1035] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06d8237bcd3548d0a51b2832e777e2bb" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:01.257053 21174 mvcc.cc:205] Tried to move back new op lower bound from 7332667642893864960 to 7332667642865664000. Current Snapshot: MvccSnapshot[applied={T|T < 7332667642893864960 or (T in {7332667642893864960})}]
I20260923 22:21:01.257794 21175 mvcc.cc:205] Tried to move back new op lower bound from 7332667642893864960 to 7332667642865664000. Current Snapshot: MvccSnapshot[applied={T|T < 7332667642893864960 or (T in {7332667642893864960})}]
I20260923 22:21:01.258816 20683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:01.258909 20683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:01.259613 20683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbcf8c6ec067404fa7bbccfeca040439: Sending DeleteTablet for 3 replicas of tablet f2b1932805bd4f7eac69d2fdf24b65be
I20260923 22:21:01.259943 20943 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b1932805bd4f7eac69d2fdf24b65be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:01 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:49272
I20260923 22:21:01.259950 21077 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b1932805bd4f7eac69d2fdf24b65be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:01 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:47662
I20260923 22:21:01.260003 20809 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b1932805bd4f7eac69d2fdf24b65be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:01 UTC) from {username='kudu', principal='kudu/127.1.129.126@KRBTEST.COM'} at 127.0.0.1:57394
I20260923 22:21:01.260149 21184 tablet_replica.cc:330] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: stopping tablet replica
I20260923 22:21:01.260229 21184 raft_consensus.cc:2229] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:21:01.260309 21184 raft_consensus.cc:2258] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:01.260317 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 20746
I20260923 22:21:01.261417 21184 ts_tablet_manager.cc:1905] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:01.263073 21184 ts_tablet_manager.cc:1918] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:01.263145 21184 log.cc:1196] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/f2b1932805bd4f7eac69d2fdf24b65be
I20260923 22:21:01.263410 21184 ts_tablet_manager.cc:1939] T f2b1932805bd4f7eac69d2fdf24b65be P f56ce126d7cb4130a4df5a8095dbff59: Deleting consensus metadata
I20260923 22:21:01.263780 20671 catalog_manager.cc:4841] TS f56ce126d7cb4130a4df5a8095dbff59 (127.1.129.67:42087): tablet f2b1932805bd4f7eac69d2fdf24b65be (table test-table [id=0934d5ad33bf4511a7cad27e9110cae8]) successfully deleted
I20260923 22:21:01.266433 21183 tablet_replica.cc:330] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28: stopping tablet replica
I20260923 22:21:01.266553 21183 raft_consensus.cc:2229] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:01.266688 21183 raft_consensus.cc:2258] T f2b1932805bd4f7eac69d2fdf24b65be P d9e666c8c6f140718b61ebac9a53ea28 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 22:21:01.266887 20671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:33639 (error 108)
I20260923 22:21:01.267124 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 20880
I20260923 22:21:01.267752 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.267269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:33639 (local address 127.0.0.1:57408)
0923 22:21:01.267675 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.267713 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:33639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
W20260923 22:21:01.267918 20671 catalog_manager.cc:4584] TS 06d8237bcd3548d0a51b2832e777e2bb (127.1.129.65:33639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2b1932805bd4f7eac69d2fdf24b65be: Network error: Client connection negotiation failed: client connection to 127.1.129.65:33639: connect: Connection refused (error 111)
I20260923 22:21:01.272734 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.272636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:40931 (local address 127.0.0.1:49274)
0923 22:21:01.272679 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.272700 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.66:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 22:21:01.272718 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 21014
I20260923 22:21:01.277664 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 20652
2026-09-23T22:21:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1906 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T22:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:01Z Disabled control of system clock
I20260923 22:21:01.304587 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34135
--webserver_interface=127.1.129.126
--webserver_port=0
--builtin_ntp_servers=127.1.129.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:01.384460 21190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.384676 21190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.384698 21190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.385766 21190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:01.385802 21190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.385818 21190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.385833 21190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:01.385846 21190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:01.387130 21190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:34135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:01.387329 21190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.387581 21190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:01.392153 21196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.392206 21198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.392302 21190 server_base.cc:1034] running on GCE node
W20260923 22:21:01.392171 21195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.392668 21190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.392943 21190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.394097 21190 hybrid_clock.cc:648] HybridClock initialized: now 1790202061394083 us; error 26 us; skew 500 ppm
I20260923 22:21:01.395397 21190 webserver.cc:460] Webserver started at http://127.1.129.126:36755/ using document root <none> and password file <none>
I20260923 22:21:01.395629 21190 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.395695 21190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.395838 21190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.396910 21190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "527ed62fd4a147ec84b51cb628422603"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "1094ab256100cc52bbd59d12fa116300"
server_key_iv: "745562975c237de0b7160845f0c73426"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.397269 21190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "527ed62fd4a147ec84b51cb628422603"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "1094ab256100cc52bbd59d12fa116300"
server_key_iv: "745562975c237de0b7160845f0c73426"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.398574 21190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:01.399317 21204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.399478 21190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:01.399570 21190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "527ed62fd4a147ec84b51cb628422603"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "1094ab256100cc52bbd59d12fa116300"
server_key_iv: "745562975c237de0b7160845f0c73426"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.399645 21190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.410104 21190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.410422 21190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.410562 21190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.414239 21190 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:34135
I20260923 22:21:01.414296 21256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:34135 every 8 connection(s)
I20260923 22:21:01.414669 21190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:01.415620 21257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:01.417908 21257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603: Bootstrap starting.
I20260923 22:21:01.418486 21257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:01.418764 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 21190
I20260923 22:21:01.418825 21257 log.cc:824] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:01.418886 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:01.419056 1541 external_mini_cluster.cc:1428] Setting key 3abe810f4b2ae67891ffb738d03b492a
I20260923 22:21:01.419545 21257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603: No bootstrap required, opened a new log
I20260923 22:21:01.421303 21257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527ed62fd4a147ec84b51cb628422603" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34135 } }
I20260923 22:21:01.421451 21257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:01.421490 21257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527ed62fd4a147ec84b51cb628422603, State: Initialized, Role: FOLLOWER
I20260923 22:21:01.421636 21257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527ed62fd4a147ec84b51cb628422603" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34135 } }
I20260923 22:21:01.421741 21257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:01.421778 21257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:01.421830 21257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:01.422354 21257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527ed62fd4a147ec84b51cb628422603" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34135 } }
I20260923 22:21:01.422466 21257 leader_election.cc:302] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 527ed62fd4a147ec84b51cb628422603; no voters:
I20260923 22:21:01.422706 21262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:01.422729 21257 leader_election.cc:288] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:01.422863 21262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [term 1 LEADER]: Becoming Leader. State: Replica: 527ed62fd4a147ec84b51cb628422603, State: Running, Role: LEADER
I20260923 22:21:01.422950 21262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527ed62fd4a147ec84b51cb628422603" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34135 } }
I20260923 22:21:01.423005 21257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:01.423249 21262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "527ed62fd4a147ec84b51cb628422603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527ed62fd4a147ec84b51cb628422603" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34135 } } }
I20260923 22:21:01.423323 21262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:01.423260 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527ed62fd4a147ec84b51cb628422603. Latest consensus state: current_term: 1 leader_uuid: "527ed62fd4a147ec84b51cb628422603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527ed62fd4a147ec84b51cb628422603" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34135 } } }
I20260923 22:21:01.423511 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:01.424130 21266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:01.424670 21266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:01.425951 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.419571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.1.129.126:34135)
0923 22:21:01.419794 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.419804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.419832 (+ 28us) server_negotiation.cc:407] Connection header received
0923 22:21:01.420662 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.420669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.420725 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.420867 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.421075 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.422469 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.423649 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.424265 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.424372 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.425301 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.425305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.425305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:01.425478 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.425509 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:01.425522 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.425594 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.425669 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.425750 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260923 22:21:01.427023 21266 catalog_manager.cc:1347] Generated new cluster ID: c18e75c0ed214e2a80a04080876546c8
I20260923 22:21:01.427080 21266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:01.433128 21266 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:01.433585 21266 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:01.440423 21266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603: Generated new TSK 0
I20260923 22:21:01.440603 21266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:01.444289 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.65:0
--local_ip_for_outbound_sockets=127.1.129.65
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34135
--builtin_ntp_servers=127.1.129.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:01.530339 21281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.530555 21281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.530591 21281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.531769 21281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.531831 21281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.531873 21281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.65
I20260923 22:21:01.533190 21281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.129.65
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.65
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:01.533430 21281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.533699 21281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:01.538545 21286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.538656 21281 server_base.cc:1034] running on GCE node
W20260923 22:21:01.538506 21287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.538522 21289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.538972 21281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.539263 21281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.540414 21281 hybrid_clock.cc:648] HybridClock initialized: now 1790202061540402 us; error 30 us; skew 500 ppm
I20260923 22:21:01.541618 21281 webserver.cc:460] Webserver started at http://127.1.129.65:46359/ using document root <none> and password file <none>
I20260923 22:21:01.541853 21281 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.541920 21281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.542063 21281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.543167 21281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/data/instance:
uuid: "72c805795d2e4afd8ede784bf0e85f98"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "a0ac3f5cade0f2f454d747a99339be7d"
server_key_iv: "f355811e2a8e4bc7d8b02ab56aa7d2a1"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.543520 21281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance:
uuid: "72c805795d2e4afd8ede784bf0e85f98"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "a0ac3f5cade0f2f454d747a99339be7d"
server_key_iv: "f355811e2a8e4bc7d8b02ab56aa7d2a1"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.544802 21281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:01.545552 21295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.545723 21281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:01.545812 21281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/wal
uuid: "72c805795d2e4afd8ede784bf0e85f98"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "a0ac3f5cade0f2f454d747a99339be7d"
server_key_iv: "f355811e2a8e4bc7d8b02ab56aa7d2a1"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.545888 21281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.559886 21281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.560194 21281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.560339 21281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.560559 21281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:01.560873 21281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:01.560925 21281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.560966 21281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:01.560992 21281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.566471 21281 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.65:38497
I20260923 22:21:01.566511 21408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.65:38497 every 8 connection(s)
I20260923 22:21:01.566874 21281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/data/info.pb
I20260923 22:21:01.569444 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 21281
I20260923 22:21:01.569566 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-0/wal/instance
I20260923 22:21:01.569801 1541 external_mini_cluster.cc:1428] Setting key 8a86157687cad8de7efd6d83b9139457
I20260923 22:21:01.570782 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.66:0
--local_ip_for_outbound_sockets=127.1.129.66
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34135
--builtin_ntp_servers=127.1.129.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 22:21:01.572549 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.567560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.65:39541 (local address 127.1.129.126:34135)
0923 22:21:01.567636 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.567638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.567828 (+ 190us) server_negotiation.cc:407] Connection header received
0923 22:21:01.568304 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.568305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.568315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.568362 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.568974 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.569449 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.570936 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.571268 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.571313 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.572258 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.572260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.572260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:01.572265 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.572282 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:01.572286 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.572346 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.572423 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.572450 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:21:01.572679 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.567537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34135 (local address 127.1.129.65:39541)
0923 22:21:01.567730 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.567763 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.568226 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.568421 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.568425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.568432 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:01.568911 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.568915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.569547 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.569549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.570826 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.570829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.571400 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.571401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.571559 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.572086 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:01.572086 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:01.572129 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:01.572168 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:01.572352 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:01.572356 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:01.572359 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.572411 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.572489 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260923 22:21:01.573101 21409 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34135
I20260923 22:21:01.573194 21409 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:01.573385 21409 heartbeater.cc:511] Master 127.1.129.126:34135 requested a full tablet report, sending...
I20260923 22:21:01.573848 21221 ts_manager.cc:194] Registered new tserver with Master: 72c805795d2e4afd8ede784bf0e85f98 (127.1.129.65:38497)
I20260923 22:21:01.574441 21221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.65:39541
W20260923 22:21:01.652786 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.653025 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.653064 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.654186 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.654249 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.654296 21412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.66
I20260923 22:21:01.655772 21412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.129.66
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.66
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:01.656038 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.656333 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:01.661365 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.661371 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.661365 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.661708 21412 server_base.cc:1034] running on GCE node
I20260923 22:21:01.661922 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.662165 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.663322 21412 hybrid_clock.cc:648] HybridClock initialized: now 1790202061663310 us; error 45 us; skew 500 ppm
I20260923 22:21:01.664630 21412 webserver.cc:460] Webserver started at http://127.1.129.66:35541/ using document root <none> and password file <none>
I20260923 22:21:01.664878 21412 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.664975 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.665129 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.666215 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/data/instance:
uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "2b46c2c9c6fb25b7ee201ea1184eaee5"
server_key_iv: "8aee9aa77d86903bfe89ed4784efa538"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.666589 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance:
uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "2b46c2c9c6fb25b7ee201ea1184eaee5"
server_key_iv: "8aee9aa77d86903bfe89ed4784efa538"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.667971 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:01.668675 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.668844 21412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:01.668964 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/wal
uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "2b46c2c9c6fb25b7ee201ea1184eaee5"
server_key_iv: "8aee9aa77d86903bfe89ed4784efa538"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.669039 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.680858 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.681154 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.681299 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.681526 21412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:01.681831 21412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:01.681882 21412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.681922 21412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:01.681958 21412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.687714 21412 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.66:39387
I20260923 22:21:01.687757 21539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.66:39387 every 8 connection(s)
I20260923 22:21:01.688109 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/data/info.pb
I20260923 22:21:01.693374 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.688836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.66:40037 (local address 127.1.129.126:34135)
0923 22:21:01.688919 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.688921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.689143 (+ 222us) server_negotiation.cc:407] Connection header received
0923 22:21:01.689638 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.689639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.689649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.689718 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.690443 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.691047 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.692092 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.692297 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.692353 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.693112 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.693113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.693114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:01.693117 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.693132 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:01.693135 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.693181 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.693293 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.693315 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 22:21:01.693535 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.688803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34135 (local address 127.1.129.66:40037)
0923 22:21:01.689017 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.689053 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.689528 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.689790 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.689793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.689802 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:01.690334 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.690338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.691122 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.691122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.691943 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.691946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.692367 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.692368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.692476 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.692966 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:01.692967 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:01.693012 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:01.693034 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:01.693190 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:01.693194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:01.693196 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.693250 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.693327 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260923 22:21:01.694133 21540 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34135
I20260923 22:21:01.694276 21540 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:01.694471 21540 heartbeater.cc:511] Master 127.1.129.126:34135 requested a full tablet report, sending...
I20260923 22:21:01.694844 21221 ts_manager.cc:194] Registered new tserver with Master: f3f4d3589c5a49cfa10dc6cbeb09caad (127.1.129.66:39387)
I20260923 22:21:01.695202 21221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.66:40037
I20260923 22:21:01.695976 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 21412
I20260923 22:21:01.696085 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/wal/instance
I20260923 22:21:01.696296 1541 external_mini_cluster.cc:1428] Setting key 016ce8e3ecd10f9dc40a348b326484cf
I20260923 22:21:01.697094 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.129.67:0
--local_ip_for_outbound_sockets=127.1.129.67
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34135
--builtin_ntp_servers=127.1.129.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:01.777236 21543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.777429 21543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.777450 21543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.778586 21543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.778651 21543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.778708 21543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.129.67
I20260923 22:21:01.780036 21543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.129.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.129.67
--webserver_port=0
--tserver_master_addrs=127.1.129.126:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.129.67
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:01.780254 21543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.780531 21543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:01.785318 21551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.785319 21548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.785326 21549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.785518 21543 server_base.cc:1034] running on GCE node
I20260923 22:21:01.785717 21543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.785974 21543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.787145 21543 hybrid_clock.cc:648] HybridClock initialized: now 1790202061787123 us; error 34 us; skew 500 ppm
I20260923 22:21:01.788489 21543 webserver.cc:460] Webserver started at http://127.1.129.67:33127/ using document root <none> and password file <none>
I20260923 22:21:01.788810 21543 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.788857 21543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.789001 21543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.790345 21543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/data/instance:
uuid: "78b8d93d381c438b9a283b0539d9a6e4"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "bab67296f81809875a6c560a510f6942"
server_key_iv: "5b450344c6b183a328a61a772c951000"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.790783 21543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance:
uuid: "78b8d93d381c438b9a283b0539d9a6e4"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "bab67296f81809875a6c560a510f6942"
server_key_iv: "5b450344c6b183a328a61a772c951000"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.792292 21543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:01.792995 21557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.793164 21543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:01.793257 21543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/wal
uuid: "78b8d93d381c438b9a283b0539d9a6e4"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-dmw7"
server_key: "bab67296f81809875a6c560a510f6942"
server_key_iv: "5b450344c6b183a328a61a772c951000"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.793336 21543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.820345 21543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.820668 21543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.820815 21543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.821029 21543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:01.821363 21543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:01.821414 21543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.821448 21543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:01.821475 21543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.827849 21543 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.67:36193
I20260923 22:21:01.827899 21670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.67:36193 every 8 connection(s)
I20260923 22:21:01.828226 21543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/data/info.pb
I20260923 22:21:01.831957 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 21543
I20260923 22:21:01.832072 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/wal/instance
I20260923 22:21:01.832275 1541 external_mini_cluster.cc:1428] Setting key 909c58bcd23223ad70467c207b254368
I20260923 22:21:01.833263 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.828895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:41255 (local address 127.1.129.126:34135)
0923 22:21:01.828941 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.828943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.829185 (+ 242us) server_negotiation.cc:407] Connection header received
0923 22:21:01.829633 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.829634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.829643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.829694 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.830300 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.830814 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.831731 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.832057 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.832128 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.833015 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.833016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.833016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:01.833020 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.833035 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:01.833038 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.833090 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.833167 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.833190 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:21:01.833456 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.828904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:34135 (local address 127.1.129.67:41255)
0923 22:21:01.829058 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.829089 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.829548 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.829744 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.829747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.829754 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:01.830225 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.830229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.830880 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.830881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.831658 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.831660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.832146 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.832147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.832287 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.832869 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:01.832869 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:01.832897 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:01.832918 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:01.833093 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:01.833097 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:01.833100 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.833155 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.833249 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260923 22:21:01.833796 21671 heartbeater.cc:347] Connected to a master server at 127.1.129.126:34135
I20260923 22:21:01.833865 21671 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:01.833989 21671 heartbeater.cc:511] Master 127.1.129.126:34135 requested a full tablet report, sending...
I20260923 22:21:01.834244 21221 ts_manager.cc:194] Registered new tserver with Master: 78b8d93d381c438b9a283b0539d9a6e4 (127.1.129.67:36193)
I20260923 22:21:01.834605 21221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.129.67:41255
I20260923 22:21:01.843772 1541 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:21:01.847509 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.845048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.1.129.126:34135)
0923 22:21:01.845100 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.845102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.845211 (+ 109us) server_negotiation.cc:407] Connection header received
0923 22:21:01.845266 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.845267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.845275 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.845312 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.845431 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.845865 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.846525 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.846747 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.846795 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.847299 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.847300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.847300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:01.847303 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.847314 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:01.847317 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.847362 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.847397 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.847414 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:21:01.849381 21221 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:21:01.851745 21221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:21:01.858675 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.854265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.1.129.67:36193)
0923 22:21:01.854407 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.854418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.854428 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:01.854467 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.854470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.854487 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.854575 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:01.854858 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.855935 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.857634 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.858294 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.858362 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.858441 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.858538 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.858580 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260923 22:21:01.858898 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.854199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.1.129.66:39387)
0923 22:21:01.854380 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.854385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.854810 (+ 425us) server_negotiation.cc:407] Connection header received
0923 22:21:01.854858 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.854860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.854877 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.854947 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:01.855150 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.856220 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.857388 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.858060 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.858130 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.858214 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.858604 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.858664 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260923 22:21:01.859062 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.853955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.1.129.65:38497)
0923 22:21:01.854471 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.854475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.854635 (+ 160us) server_negotiation.cc:407] Connection header received
0923 22:21:01.854947 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.854949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.854964 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.855036 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:01.855249 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.856302 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.857982 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.858570 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.858628 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.858741 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.858933 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.858984 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":94,"threads_started":1}
I20260923 22:21:01.858682 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.853865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.67:36193 (local address 127.0.0.1:38498)
0923 22:21:01.854169 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.854211 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.854277 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.854705 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.854709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.854711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:01.854773 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.854774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.856024 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.856025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.857555 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.857558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.858374 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.858376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.858437 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.858440 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.858517 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.858559 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":73,"threads_started":1}
I20260923 22:21:01.858548 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.854066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:39387 (local address 127.0.0.1:43564)
0923 22:21:01.854676 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.854687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.854743 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.855018 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.855019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.855021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:01.855066 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.855067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.856322 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.856323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.857298 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.857300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.858249 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.858250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.858325 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.858341 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.858402 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.858464 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":98,"threads_started":1}
I20260923 22:21:01.859025 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.853880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:38497 (local address 127.0.0.1:42188)
0923 22:21:01.854526 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.854535 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.854581 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.855103 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.855104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.855106 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:01.855145 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.855147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.856386 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.856387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.857909 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.857912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.858782 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.858783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.858846 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.858849 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.858903 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.858943 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":66,"threads_started":1}
I20260923 22:21:01.859793 21604 tablet_service.cc:1467] Processing CreateTablet for tablet 714b62af929f4b518165ce8af7861a58 (DEFAULT_TABLE table=test-table [id=1fa2a1585a49495db3e66a7215972f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:01.859907 21474 tablet_service.cc:1467] Processing CreateTablet for tablet 714b62af929f4b518165ce8af7861a58 (DEFAULT_TABLE table=test-table [id=1fa2a1585a49495db3e66a7215972f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:01.860152 21604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714b62af929f4b518165ce8af7861a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:01.860206 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714b62af929f4b518165ce8af7861a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:01.860119 21343 tablet_service.cc:1467] Processing CreateTablet for tablet 714b62af929f4b518165ce8af7861a58 (DEFAULT_TABLE table=test-table [id=1fa2a1585a49495db3e66a7215972f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:01.860455 21343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714b62af929f4b518165ce8af7861a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:01.862980 21685 tablet_bootstrap.cc:492] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98: Bootstrap starting.
I20260923 22:21:01.863358 21686 tablet_bootstrap.cc:492] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Bootstrap starting.
I20260923 22:21:01.863833 21685 tablet_bootstrap.cc:654] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:01.863941 21686 tablet_bootstrap.cc:654] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:01.864197 21685 log.cc:824] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:01.864245 21686 log.cc:824] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:01.864543 21687 tablet_bootstrap.cc:492] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Bootstrap starting.
I20260923 22:21:01.865026 21686 tablet_bootstrap.cc:492] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: No bootstrap required, opened a new log
I20260923 22:21:01.865101 21685 tablet_bootstrap.cc:492] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98: No bootstrap required, opened a new log
I20260923 22:21:01.865166 21687 tablet_bootstrap.cc:654] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:01.865175 21685 ts_tablet_manager.cc:1397] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:01.865341 21686 ts_tablet_manager.cc:1397] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:21:01.865499 21687 log.cc:824] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:01.866340 21687 tablet_bootstrap.cc:492] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: No bootstrap required, opened a new log
I20260923 22:21:01.866405 21687 ts_tablet_manager.cc:1397] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:01.867270 21685 raft_consensus.cc:348] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:01.867411 21685 raft_consensus.cc:374] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:01.867452 21685 raft_consensus.cc:729] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c805795d2e4afd8ede784bf0e85f98, State: Initialized, Role: FOLLOWER
I20260923 22:21:01.867569 21685 consensus_queue.cc:260] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:01.867815 21685 ts_tablet_manager.cc:1428] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:01.867933 21409 heartbeater.cc:503] Master 127.1.129.126:34135 was elected leader, sending a full tablet report...
I20260923 22:21:01.867990 21686 raft_consensus.cc:348] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:01.868165 21686 raft_consensus.cc:374] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:01.868202 21686 raft_consensus.cc:729] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f4d3589c5a49cfa10dc6cbeb09caad, State: Initialized, Role: FOLLOWER
I20260923 22:21:01.868322 21686 consensus_queue.cc:260] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:01.868584 21686 ts_tablet_manager.cc:1428] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:01.868629 21540 heartbeater.cc:503] Master 127.1.129.126:34135 was elected leader, sending a full tablet report...
I20260923 22:21:01.869122 21687 raft_consensus.cc:348] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:01.869302 21687 raft_consensus.cc:374] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:01.869340 21687 raft_consensus.cc:729] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b8d93d381c438b9a283b0539d9a6e4, State: Initialized, Role: FOLLOWER
I20260923 22:21:01.869457 21687 consensus_queue.cc:260] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:01.869706 21687 ts_tablet_manager.cc:1428] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:01.869773 21671 heartbeater.cc:503] Master 127.1.129.126:34135 was elected leader, sending a full tablet report...
W20260923 22:21:01.938711 21541 tablet.cc:2369] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:21:02.067622 21410 tablet.cc:2369] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:21:02.078847 21672 tablet.cc:2369] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:21:02.232852 21693 raft_consensus.cc:482] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:21:02.233053 21693 raft_consensus.cc:504] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:02.233451 21693 leader_election.cc:288] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72c805795d2e4afd8ede784bf0e85f98 (127.1.129.65:38497), f3f4d3589c5a49cfa10dc6cbeb09caad (127.1.129.66:39387)
I20260923 22:21:02.237411 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.233770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:47973 (local address 127.1.129.66:39387)
0923 22:21:02.233825 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.233827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.233966 (+ 139us) server_negotiation.cc:407] Connection header received
0923 22:21:02.234027 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.234028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.234039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.234090 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.234248 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.234961 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.236531 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.237067 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.237141 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.237149 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.237303 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.237339 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:21:02.237411 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.233735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.129.67:43779 (local address 127.1.129.65:38497)
0923 22:21:02.233783 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.233786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.233798 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:21:02.233901 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.233902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.233913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.233983 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.234181 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.234961 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.236531 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.237073 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.237141 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.237149 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.237303 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.237340 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:21:02.237448 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.233638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:38497 (local address 127.1.129.67:43779)
0923 22:21:02.233708 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.233723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.233795 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.234006 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.234007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.234008 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.234100 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.234102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.235061 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.235063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.236436 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.236440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.237147 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.237147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.237225 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.237231 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.237285 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.237333 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":121}
I20260923 22:21:02.237634 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.233638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.66:39387 (local address 127.1.129.67:47973)
0923 22:21:02.233879 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.233889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.233951 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.234107 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.234108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.234110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.234176 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.234178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.235061 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.235063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.236446 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.236448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.237147 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.237148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.237225 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.237231 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.237285 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.237331 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":144,"thread_start_us":99,"threads_started":1}
I20260923 22:21:02.237874 21363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714b62af929f4b518165ce8af7861a58" candidate_uuid: "78b8d93d381c438b9a283b0539d9a6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c805795d2e4afd8ede784bf0e85f98" is_pre_election: true
I20260923 22:21:02.238075 21363 raft_consensus.cc:2454] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b8d93d381c438b9a283b0539d9a6e4 in term 0.
I20260923 22:21:02.238178 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714b62af929f4b518165ce8af7861a58" candidate_uuid: "78b8d93d381c438b9a283b0539d9a6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" is_pre_election: true
I20260923 22:21:02.238348 21494 raft_consensus.cc:2454] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b8d93d381c438b9a283b0539d9a6e4 in term 0.
I20260923 22:21:02.238418 21560 leader_election.cc:302] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c805795d2e4afd8ede784bf0e85f98, 78b8d93d381c438b9a283b0539d9a6e4; no voters:
I20260923 22:21:02.238544 21693 raft_consensus.cc:2781] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:21:02.238595 21693 raft_consensus.cc:482] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:21:02.238615 21693 raft_consensus.cc:3037] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:02.239367 21693 raft_consensus.cc:504] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:02.239603 21693 leader_election.cc:288] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [CANDIDATE]: Term 1 election: Requested vote from peers 72c805795d2e4afd8ede784bf0e85f98 (127.1.129.65:38497), f3f4d3589c5a49cfa10dc6cbeb09caad (127.1.129.66:39387)
I20260923 22:21:02.239679 21363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714b62af929f4b518165ce8af7861a58" candidate_uuid: "78b8d93d381c438b9a283b0539d9a6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c805795d2e4afd8ede784bf0e85f98"
I20260923 22:21:02.239755 21363 raft_consensus.cc:3037] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:02.239741 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714b62af929f4b518165ce8af7861a58" candidate_uuid: "78b8d93d381c438b9a283b0539d9a6e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad"
I20260923 22:21:02.239825 21494 raft_consensus.cc:3037] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:02.240603 21363 raft_consensus.cc:2454] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b8d93d381c438b9a283b0539d9a6e4 in term 1.
I20260923 22:21:02.240741 21494 raft_consensus.cc:2454] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b8d93d381c438b9a283b0539d9a6e4 in term 1.
I20260923 22:21:02.240794 21560 leader_election.cc:302] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c805795d2e4afd8ede784bf0e85f98, 78b8d93d381c438b9a283b0539d9a6e4; no voters:
I20260923 22:21:02.240885 21693 raft_consensus.cc:2781] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:02.241022 21693 raft_consensus.cc:686] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 1 LEADER]: Becoming Leader. State: Replica: 78b8d93d381c438b9a283b0539d9a6e4, State: Running, Role: LEADER
I20260923 22:21:02.241125 21693 consensus_queue.cc:237] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } }
I20260923 22:21:02.241948 21220 catalog_manager.cc:5495] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b8d93d381c438b9a283b0539d9a6e4 (127.1.129.67). New cstate: current_term: 1 leader_uuid: "78b8d93d381c438b9a283b0539d9a6e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b8d93d381c438b9a283b0539d9a6e4" member_type: VOTER last_known_addr { host: "127.1.129.67" port: 36193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 } health_report { overall_health: UNKNOWN } } }
I20260923 22:21:02.297467 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.294907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.1.129.67:36193)
0923 22:21:02.294967 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.294969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.294995 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:21:02.295045 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.295045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.295062 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.295149 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:02.295290 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.295806 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.296672 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.296879 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.296952 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.297133 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:02.297233 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:02.297319 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.297342 (+ 23us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.297361 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:21:02.299175 21494 raft_consensus.cc:1261] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 1 FOLLOWER]: Refusing update from remote peer 78b8d93d381c438b9a283b0539d9a6e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:02.299222 21363 raft_consensus.cc:1261] T 714b62af929f4b518165ce8af7861a58 P 72c805795d2e4afd8ede784bf0e85f98 [term 1 FOLLOWER]: Refusing update from remote peer 78b8d93d381c438b9a283b0539d9a6e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:02.299576 21693 consensus_queue.cc:1035] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f4d3589c5a49cfa10dc6cbeb09caad" member_type: VOTER last_known_addr { host: "127.1.129.66" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:02.299757 21695 consensus_queue.cc:1035] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c805795d2e4afd8ede784bf0e85f98" member_type: VOTER last_known_addr { host: "127.1.129.65" port: 38497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:02.302402 21700 mvcc.cc:205] Tried to move back new op lower bound from 7332667647174758400 to 7332667646940160000. Current Snapshot: MvccSnapshot[applied={T|T < 7332667647174758400}]
I20260923 22:21:02.303977 21220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:02.304075 21220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:02.304770 21220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 527ed62fd4a147ec84b51cb628422603: Sending DeleteTablet for 3 replicas of tablet 714b62af929f4b518165ce8af7861a58
I20260923 22:21:02.305104 21343 tablet_service.cc:1514] Processing DeleteTablet for tablet 714b62af929f4b518165ce8af7861a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:02 UTC) from {username='slave'} at 127.0.0.1:42188
I20260923 22:21:02.305146 21604 tablet_service.cc:1514] Processing DeleteTablet for tablet 714b62af929f4b518165ce8af7861a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:02 UTC) from {username='slave'} at 127.0.0.1:38498
I20260923 22:21:02.305131 21474 tablet_service.cc:1514] Processing DeleteTablet for tablet 714b62af929f4b518165ce8af7861a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:02 UTC) from {username='slave'} at 127.0.0.1:43564
I20260923 22:21:02.305332 21710 tablet_replica.cc:330] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: stopping tablet replica
I20260923 22:21:02.305414 21710 raft_consensus.cc:2229] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:02.305526 21710 raft_consensus.cc:2258] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:02.305550 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 21281
I20260923 22:21:02.305949 21710 ts_tablet_manager.cc:1905] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:02.307615 21710 ts_tablet_manager.cc:1918] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:02.307701 21710 log.cc:1196] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-1/wal/wals/714b62af929f4b518165ce8af7861a58
I20260923 22:21:02.307957 21710 ts_tablet_manager.cc:1939] T 714b62af929f4b518165ce8af7861a58 P f3f4d3589c5a49cfa10dc6cbeb09caad: Deleting consensus metadata
I20260923 22:21:02.308378 21205 catalog_manager.cc:4841] TS f3f4d3589c5a49cfa10dc6cbeb09caad (127.1.129.66:39387): tablet 714b62af929f4b518165ce8af7861a58 (table test-table [id=1fa2a1585a49495db3e66a7215972f09]) successfully deleted
I20260923 22:21:02.310779 21709 tablet_replica.cc:330] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: stopping tablet replica
I20260923 22:21:02.310904 21709 raft_consensus.cc:2229] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:21:02.311013 21709 raft_consensus.cc:2258] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:02.311425 21709 ts_tablet_manager.cc:1905] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 22:21:02.312227 21207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.129.65:38497 (error 108)
I20260923 22:21:02.312734 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.312601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.65:38497 (local address 127.0.0.1:42198)
0923 22:21:02.312647 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.312693 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.65:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 22:21:02.312822 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 21412
W20260923 22:21:02.312855 21207 catalog_manager.cc:4584] TS 72c805795d2e4afd8ede784bf0e85f98 (127.1.129.65:38497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 714b62af929f4b518165ce8af7861a58: Network error: Client connection negotiation failed: client connection to 127.1.129.65:38497: connect: Connection refused (error 111)
I20260923 22:21:02.313376 21709 ts_tablet_manager.cc:1918] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:02.313444 21709 log.cc:1196] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Deleting WAL directory at /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201819212015-1541-0/minicluster-data/ts-2/wal/wals/714b62af929f4b518165ce8af7861a58
I20260923 22:21:02.313696 21709 ts_tablet_manager.cc:1939] T 714b62af929f4b518165ce8af7861a58 P 78b8d93d381c438b9a283b0539d9a6e4: Deleting consensus metadata
I20260923 22:21:02.314064 21205 catalog_manager.cc:4841] TS 78b8d93d381c438b9a283b0539d9a6e4 (127.1.129.67:36193): tablet 714b62af929f4b518165ce8af7861a58 (table test-table [id=1fa2a1585a49495db3e66a7215972f09]) successfully deleted
I20260923 22:21:02.317821 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 21543
I20260923 22:21:02.322463 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 21190
2026-09-23T22:21:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1044 ms)
[----------] 6 tests from EncryptionPolicyTest (8560 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:02 dist-test-slave-dmw7 krb5kdc[21714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:02 dist-test-slave-dmw7 krb5kdc[21714](info): set up 2 sockets
Sep 23 22:21:02 dist-test-slave-dmw7 krb5kdc[21714](info): commencing operation
krb5kdc: starting...
W20260923 22:21:03.358763 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:03 dist-test-slave-dmw7 krb5kdc[21714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:03.476994 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:03.558044 21727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:03.558308 21727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:03.558346 21727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:03.559540 21727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:03.559612 21727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:03.559635 21727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:03.559659 21727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:03.559690 21727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:03.561196 21727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:03.561474 21727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:03.561697 21727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:21:03.566558 21727 server_base.cc:1034] running on GCE node
W20260923 22:21:03.566551 21733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:03.566529 21735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:03.566532 21732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:03.567065 21727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:03.567375 21727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:03.568535 21727 hybrid_clock.cc:648] HybridClock initialized: now 1790202063568529 us; error 34 us; skew 500 ppm
Sep 23 22:21:03 dist-test-slave-dmw7 krb5kdc[21714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:03.576099 21727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:03.576550 21727 webserver.cc:460] Webserver started at http://127.0.0.1:42233/ using document root <none> and password file <none>
I20260923 22:21:03.576740 21727 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:03.576778 21727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:03.576866 21727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:03.577860 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "f29902c785aa4a2980e244f93e40a992"
format_stamp: "Formatted at 2026-09-23 22:21:03 on dist-test-slave-dmw7"
server_key: "cadf537d90e8c7b4a25c955fa516f37b"
server_key_iv: "de2c9a85c1e0dd16766b827e06ddb43d"
server_key_version: "encryptionkey@0"
I20260923 22:21:03.578164 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "f29902c785aa4a2980e244f93e40a992"
format_stamp: "Formatted at 2026-09-23 22:21:03 on dist-test-slave-dmw7"
server_key: "cadf537d90e8c7b4a25c955fa516f37b"
server_key_iv: "de2c9a85c1e0dd16766b827e06ddb43d"
server_key_version: "encryptionkey@0"
I20260923 22:21:03.579517 21727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:03.580258 21742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:03.580440 21727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:03.580516 21727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "f29902c785aa4a2980e244f93e40a992"
format_stamp: "Formatted at 2026-09-23 22:21:03 on dist-test-slave-dmw7"
server_key: "cadf537d90e8c7b4a25c955fa516f37b"
server_key_iv: "de2c9a85c1e0dd16766b827e06ddb43d"
server_key_version: "encryptionkey@0"
I20260923 22:21:03.580567 21727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:03.603680 21727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:03.605589 21727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:03.605701 21727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:03.609623 21727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37071
I20260923 22:21:03.609655 21794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37071 every 8 connection(s)
I20260923 22:21:03.609997 21727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:03.610997 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:03.612181 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 21727
I20260923 22:21:03.612270 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:03.612442 1541 external_mini_cluster.cc:1428] Setting key e0f57957bac2ed9e8876bf758f3cd951
I20260923 22:21:03.613781 21795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992: Bootstrap starting.
I20260923 22:21:03.614396 21795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:03.614742 21795 log.cc:824] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:03.615378 21795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992: No bootstrap required, opened a new log
I20260923 22:21:03.617694 21795 raft_consensus.cc:348] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29902c785aa4a2980e244f93e40a992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260923 22:21:03.617867 21795 raft_consensus.cc:374] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:03.617905 21795 raft_consensus.cc:729] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29902c785aa4a2980e244f93e40a992, State: Initialized, Role: FOLLOWER
I20260923 22:21:03.618040 21795 consensus_queue.cc:260] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29902c785aa4a2980e244f93e40a992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260923 22:21:03.618132 21795 raft_consensus.cc:388] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:03.618167 21795 raft_consensus.cc:482] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:03.618219 21795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:03.618947 21795 raft_consensus.cc:504] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29902c785aa4a2980e244f93e40a992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260923 22:21:03.619031 21795 leader_election.cc:302] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f29902c785aa4a2980e244f93e40a992; no voters:
I20260923 22:21:03.619202 21795 leader_election.cc:288] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:03.619247 21800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 1 FOLLOWER]: Leader election won for term 1
Sep 23 22:21:03 dist-test-slave-dmw7 krb5kdc[21714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 22:21:03.619390 21800 raft_consensus.cc:686] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [term 1 LEADER]: Becoming Leader. State: Replica: f29902c785aa4a2980e244f93e40a992, State: Running, Role: LEADER
I20260923 22:21:03.619464 21795 sys_catalog.cc:564] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:03.619483 21800 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29902c785aa4a2980e244f93e40a992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260923 22:21:03.619778 21802 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29902c785aa4a2980e244f93e40a992. Latest consensus state: current_term: 1 leader_uuid: "f29902c785aa4a2980e244f93e40a992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29902c785aa4a2980e244f93e40a992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } } }
I20260923 22:21:03.619863 21802 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:03.620014 21801 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f29902c785aa4a2980e244f93e40a992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29902c785aa4a2980e244f93e40a992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } } }
I20260923 22:21:03.620070 21801 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:03.620187 21805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:03.620682 21805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:03.622717 21805 catalog_manager.cc:1347] Generated new cluster ID: c3c0e427ab36415ea424abee659def21
I20260923 22:21:03.622766 21805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:03.623872 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:03.612946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47509 (local address 127.0.0.1:37071)
0923 22:21:03.613134 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:03.613143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:03.613185 (+ 42us) server_negotiation.cc:407] Connection header received
0923 22:21:03.613723 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:03.613732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:03.613793 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:03.613938 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:03.614560 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.615601 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.616650 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.617030 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.617103 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:03.620058 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:03.620063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:03.620063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:03.620078 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:03.622118 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:03.622613 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:03.622616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:03.622616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:03.622759 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:03.623012 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:03.623013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:03.623013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:03.623294 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:03.623429 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:03.623572 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 22:21:03.623693 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260923 22:21:03.624851 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:03.630514 21805 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:03.631148 21805 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:03.633746 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:03.626291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38821 (local address 127.0.0.1:37071)
0923 22:21:03.626330 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:03.626332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:03.626466 (+ 134us) server_negotiation.cc:407] Connection header received
0923 22:21:03.626514 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:03.626515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:03.626524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:03.626575 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:03.627192 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.628193 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.629199 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.629510 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.629574 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:03.630975 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:03.630977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:03.630977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:03.630984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:03.632532 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:03.632957 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:03.632958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:03.632959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:03.633012 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:03.633241 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:03.633243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:03.633243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:03.633367 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:03.633457 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:03.633604 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 22:21:03.633663 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:21:03.637022 21805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f29902c785aa4a2980e244f93e40a992: Generated new TSK 0
I20260923 22:21:03.637198 21805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:03.638981 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 21727
W20260923 22:21:03.643868 21712 connection.cc:535] client connection to 127.0.0.1:37071 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37071): Connection reset by peer (error 104)
2026-09-23T22:21:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:03 dist-test-slave-dmw7 krb5kdc[21826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:03 dist-test-slave-dmw7 krb5kdc[21826](info): set up 2 sockets
Sep 23 22:21:03 dist-test-slave-dmw7 krb5kdc[21826](info): commencing operation
krb5kdc: starting...
W20260923 22:21:04.674934 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:04 dist-test-slave-dmw7 krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:04.786805 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:04.867326 21839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:04.867527 21839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:04.867551 21839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:04.868692 21839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:04.868737 21839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:04.868753 21839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:04.868768 21839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:04.868795 21839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:04.870268 21839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:04.870536 21839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:04.870814 21839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:04.875586 21845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.875586 21847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.875635 21844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:04.875902 21839 server_base.cc:1034] running on GCE node
I20260923 22:21:04.876133 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:04.876399 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:04.877573 21839 hybrid_clock.cc:648] HybridClock initialized: now 1790202064877557 us; error 25 us; skew 500 ppm
Sep 23 22:21:04 dist-test-slave-dmw7 krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:04.883291 21839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:04.883764 21839 webserver.cc:460] Webserver started at http://127.0.0.1:35245/ using document root <none> and password file <none>
I20260923 22:21:04.884008 21839 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:04.884074 21839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:04.884222 21839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:04.885247 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "629ceca6f44145f3a10369ebec6361be"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-dmw7"
server_key: "0ab9245734fa961495c0720f4ad189b3"
server_key_iv: "094e86fcf408436d3ea553855509007c"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.885581 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "629ceca6f44145f3a10369ebec6361be"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-dmw7"
server_key: "0ab9245734fa961495c0720f4ad189b3"
server_key_iv: "094e86fcf408436d3ea553855509007c"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.886906 21839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:21:04.887585 21854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.887744 21839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.887840 21839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "629ceca6f44145f3a10369ebec6361be"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-dmw7"
server_key: "0ab9245734fa961495c0720f4ad189b3"
server_key_iv: "094e86fcf408436d3ea553855509007c"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.887908 21839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:04.899683 21839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:04.901787 21839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:04.901943 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:04.905977 21839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41857
I20260923 22:21:04.906013 21906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41857 every 8 connection(s)
I20260923 22:21:04.906363 21839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:04.907336 21907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:04.909490 21907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be: Bootstrap starting.
I20260923 22:21:04.909999 21907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:04.910233 21907 log.cc:824] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:04.910898 21907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be: No bootstrap required, opened a new log
I20260923 22:21:04.910971 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 21839
I20260923 22:21:04.911100 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:04.911281 1541 external_mini_cluster.cc:1428] Setting key 20930e7d1ed0bc3ebfea582560fba399
I20260923 22:21:04.912809 21907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629ceca6f44145f3a10369ebec6361be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } }
I20260923 22:21:04.912979 21907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:04.913029 21907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629ceca6f44145f3a10369ebec6361be, State: Initialized, Role: FOLLOWER
I20260923 22:21:04.913170 21907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629ceca6f44145f3a10369ebec6361be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } }
I20260923 22:21:04.913260 21907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:04.913311 21907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:04.913364 21907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:04.913893 21907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629ceca6f44145f3a10369ebec6361be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } }
I20260923 22:21:04.913978 21907 leader_election.cc:302] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 629ceca6f44145f3a10369ebec6361be; no voters:
I20260923 22:21:04.914167 21907 leader_election.cc:288] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:04.914182 21912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:04.914296 21912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [term 1 LEADER]: Becoming Leader. State: Replica: 629ceca6f44145f3a10369ebec6361be, State: Running, Role: LEADER
I20260923 22:21:04.914381 21912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629ceca6f44145f3a10369ebec6361be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } }
I20260923 22:21:04.914425 21907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:04.914718 21913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "629ceca6f44145f3a10369ebec6361be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629ceca6f44145f3a10369ebec6361be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } } }
I20260923 22:21:04.914953 21913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:04.914875 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 629ceca6f44145f3a10369ebec6361be. Latest consensus state: current_term: 1 leader_uuid: "629ceca6f44145f3a10369ebec6361be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629ceca6f44145f3a10369ebec6361be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } } }
I20260923 22:21:04.915216 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:04.915545 21921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:04.916025 21921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:04.917832 21921 catalog_manager.cc:1347] Generated new cluster ID: 1fc66538d580426fb69b918afe36a18d
I20260923 22:21:04.917882 21921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:04.918111 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.911758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60349 (local address 127.0.0.1:41857)
0923 22:21:04.911924 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.911932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.911959 (+ 27us) server_negotiation.cc:407] Connection header received
0923 22:21:04.912788 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.912794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.912838 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.912945 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:04.913211 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.914731 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.915950 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.916529 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.916653 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.917369 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:04.917372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:04.917373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:04.917573 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:04.917610 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:04.917624 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:04.917722 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.917893 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.917920 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260923 22:21:04.919512 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:04.925769 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.922951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38187 (local address 127.0.0.1:41857)
0923 22:21:04.922989 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.922991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.923110 (+ 119us) server_negotiation.cc:407] Connection header received
0923 22:21:04.923169 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.923170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.923182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.923232 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:04.923377 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.923837 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.924533 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.924757 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.924833 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.925492 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:04.925493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:04.925493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:04.925498 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:04.925512 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:04.925516 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:04.925582 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.925650 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.925671 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:04.933223 21921 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:04.933951 21921 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:04.947919 21921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 629ceca6f44145f3a10369ebec6361be: Generated new TSK 0
I20260923 22:21:04.948134 21921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:04.953303 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 21839
W20260923 22:21:04.957607 21824 connection.cc:535] client connection to 127.0.0.1:41857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41857): Connection reset by peer (error 104)
2026-09-23T22:21:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:05 dist-test-slave-dmw7 krb5kdc[21938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:05 dist-test-slave-dmw7 krb5kdc[21938](info): set up 2 sockets
Sep 23 22:21:05 dist-test-slave-dmw7 krb5kdc[21938](info): commencing operation
krb5kdc: starting...
W20260923 22:21:05.978710 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:06 dist-test-slave-dmw7 krb5kdc[21938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:06.089650 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:06.167152 21951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:06.167397 21951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:06.167434 21951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:06.168617 21951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:06.168695 21951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:06.168720 21951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:06.168740 21951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:06.168768 21951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:06.170192 21951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:06.170451 21951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:06.170712 21951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:06.175298 21959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:06.175318 21957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:06.175330 21956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:06.175621 21951 server_base.cc:1034] running on GCE node
I20260923 22:21:06.175835 21951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:06.176090 21951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:06.177256 21951 hybrid_clock.cc:648] HybridClock initialized: now 1790202066177254 us; error 41 us; skew 500 ppm
Sep 23 22:21:06 dist-test-slave-dmw7 krb5kdc[21938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:06.179711 21951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:06.180141 21951 webserver.cc:460] Webserver started at http://127.0.0.1:36911/ using document root <none> and password file <none>
I20260923 22:21:06.180352 21951 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:06.180397 21951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:06.180552 21951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:06.181533 21951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "c9f9831eb5e64ff89cd0de229fae973a"
format_stamp: "Formatted at 2026-09-23 22:21:06 on dist-test-slave-dmw7"
server_key: "7292197ab18e346dc72e150fe861045a"
server_key_iv: "7e57c364330d946d426f74be72de293f"
server_key_version: "encryptionkey@0"
I20260923 22:21:06.181881 21951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "c9f9831eb5e64ff89cd0de229fae973a"
format_stamp: "Formatted at 2026-09-23 22:21:06 on dist-test-slave-dmw7"
server_key: "7292197ab18e346dc72e150fe861045a"
server_key_iv: "7e57c364330d946d426f74be72de293f"
server_key_version: "encryptionkey@0"
I20260923 22:21:06.183179 21951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:06.183846 21966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:06.184015 21951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:06.184098 21951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "c9f9831eb5e64ff89cd0de229fae973a"
format_stamp: "Formatted at 2026-09-23 22:21:06 on dist-test-slave-dmw7"
server_key: "7292197ab18e346dc72e150fe861045a"
server_key_iv: "7e57c364330d946d426f74be72de293f"
server_key_version: "encryptionkey@0"
I20260923 22:21:06.184170 21951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:06.187337 21951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:06.189432 21951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:06.189574 21951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:06.193326 21951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41343
I20260923 22:21:06.193354 22018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41343 every 8 connection(s)
I20260923 22:21:06.193658 21951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:06.194489 22019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:06.196646 22019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a: Bootstrap starting.
I20260923 22:21:06.197176 22019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:06.197439 22019 log.cc:824] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:06.198071 22019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a: No bootstrap required, opened a new log
I20260923 22:21:06.199649 22019 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f9831eb5e64ff89cd0de229fae973a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41343 } }
I20260923 22:21:06.199772 22019 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:06.199815 22019 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f9831eb5e64ff89cd0de229fae973a, State: Initialized, Role: FOLLOWER
I20260923 22:21:06.199980 22019 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f9831eb5e64ff89cd0de229fae973a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41343 } }
I20260923 22:21:06.200059 22019 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:06.200104 22019 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:06.200152 22019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:06.200636 22019 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f9831eb5e64ff89cd0de229fae973a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41343 } }
I20260923 22:21:06.200742 22019 leader_election.cc:302] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9f9831eb5e64ff89cd0de229fae973a; no voters:
I20260923 22:21:06.200907 22019 leader_election.cc:288] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:06.200937 22022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:06.201086 22022 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [term 1 LEADER]: Becoming Leader. State: Replica: c9f9831eb5e64ff89cd0de229fae973a, State: Running, Role: LEADER
I20260923 22:21:06.201138 22019 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:06.201238 22022 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f9831eb5e64ff89cd0de229fae973a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41343 } }
I20260923 22:21:06.201601 22023 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9f9831eb5e64ff89cd0de229fae973a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f9831eb5e64ff89cd0de229fae973a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41343 } } }
I20260923 22:21:06.201608 22024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9f9831eb5e64ff89cd0de229fae973a. Latest consensus state: current_term: 1 leader_uuid: "c9f9831eb5e64ff89cd0de229fae973a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f9831eb5e64ff89cd0de229fae973a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41343 } } }
I20260923 22:21:06.201753 22024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:06.201921 22023 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:06.202080 22026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:06.202555 22026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:06.203809 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 21951
I20260923 22:21:06.203918 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:06.204093 1541 external_mini_cluster.cc:1428] Setting key 58b833509ba41e47ed043f25c24b2e70
I20260923 22:21:06.204125 22026 catalog_manager.cc:1347] Generated new cluster ID: 3b23f59061614ec9a0c81f9fb1dbbf39
I20260923 22:21:06.204172 22026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:06.208218 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:06.204521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37073 (local address 127.0.0.1:41343)
0923 22:21:06.204678 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:06.204687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:06.206902 (+ 2215us) server_negotiation.cc:407] Connection header received
0923 22:21:06.207054 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:06.207059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:06.207082 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:06.207209 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:06.207798 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:06.207802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:06.207802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:06.207806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:06.207841 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:06.207857 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:06.207918 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:06.207998 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 22:21:06.208047 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260923 22:21:06.209096 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:06.211722 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:06.210405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53065 (local address 127.0.0.1:41343)
0923 22:21:06.210441 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:06.210443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:06.210789 (+ 346us) server_negotiation.cc:407] Connection header received
0923 22:21:06.210862 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:06.210863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:06.210869 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:06.210927 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:06.211489 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:06.211490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:06.211490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:06.211495 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:06.211509 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:06.211513 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:06.211564 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:06.211629 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 22:21:06.211642 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:06.217289 22026 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:06.217449 22026 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:06.228071 22026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9f9831eb5e64ff89cd0de229fae973a: Generated new TSK 0
I20260923 22:21:06.228266 22026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:06.235448 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 21951
2026-09-23T22:21:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:06 dist-test-slave-dmw7 krb5kdc[22050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:06 dist-test-slave-dmw7 krb5kdc[22050](info): set up 2 sockets
Sep 23 22:21:06 dist-test-slave-dmw7 krb5kdc[22050](info): commencing operation
krb5kdc: starting...
W20260923 22:21:07.254760 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:07 dist-test-slave-dmw7 krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:07.371408 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:07.452924 22063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:07.453188 22063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:07.453228 22063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:07.454391 22063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:07.454470 22063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:07.454499 22063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:07.454532 22063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:07.454562 22063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:07.456068 22063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:07.456342 22063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:07.456576 22063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:07.461548 22068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:07.461519 22069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:07.461535 22071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:07.461790 22063 server_base.cc:1034] running on GCE node
I20260923 22:21:07.462088 22063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:07.462378 22063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:07.463541 22063 hybrid_clock.cc:648] HybridClock initialized: now 1790202067463525 us; error 28 us; skew 500 ppm
Sep 23 22:21:07 dist-test-slave-dmw7 krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:07.466035 22063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:07.466475 22063 webserver.cc:460] Webserver started at http://127.0.0.1:39389/ using document root <none> and password file <none>
I20260923 22:21:07.466732 22063 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:07.466778 22063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:07.466876 22063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:07.467885 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "c85988eb857c4869a60f5e64f0ff1d58"
format_stamp: "Formatted at 2026-09-23 22:21:07 on dist-test-slave-dmw7"
server_key: "bcc2de795f78139ae126a2ca7fc635e6"
server_key_iv: "1d1b66ac2ec2abff3f3d41456d82c604"
server_key_version: "encryptionkey@0"
I20260923 22:21:07.468191 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "c85988eb857c4869a60f5e64f0ff1d58"
format_stamp: "Formatted at 2026-09-23 22:21:07 on dist-test-slave-dmw7"
server_key: "bcc2de795f78139ae126a2ca7fc635e6"
server_key_iv: "1d1b66ac2ec2abff3f3d41456d82c604"
server_key_version: "encryptionkey@0"
I20260923 22:21:07.469399 22063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:07.470019 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:07.470196 22063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:07.470268 22063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "c85988eb857c4869a60f5e64f0ff1d58"
format_stamp: "Formatted at 2026-09-23 22:21:07 on dist-test-slave-dmw7"
server_key: "bcc2de795f78139ae126a2ca7fc635e6"
server_key_iv: "1d1b66ac2ec2abff3f3d41456d82c604"
server_key_version: "encryptionkey@0"
I20260923 22:21:07.470319 22063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:07.485405 22063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:07.487722 22063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:07.487831 22063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:07.491680 22063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35791
I20260923 22:21:07.491735 22130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35791 every 8 connection(s)
I20260923 22:21:07.492079 22063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:07.493016 22131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:07.495250 22131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58: Bootstrap starting.
I20260923 22:21:07.495749 22131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:07.495973 22131 log.cc:824] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:07.496248 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22063
I20260923 22:21:07.496361 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:07.496544 1541 external_mini_cluster.cc:1428] Setting key 96e8f453755239b0cb0c88e055ec1fcc
I20260923 22:21:07.496742 22131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58: No bootstrap required, opened a new log
I20260923 22:21:07.498525 22131 raft_consensus.cc:348] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85988eb857c4869a60f5e64f0ff1d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35791 } }
I20260923 22:21:07.498701 22131 raft_consensus.cc:374] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:07.498736 22131 raft_consensus.cc:729] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85988eb857c4869a60f5e64f0ff1d58, State: Initialized, Role: FOLLOWER
I20260923 22:21:07.498868 22131 consensus_queue.cc:260] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85988eb857c4869a60f5e64f0ff1d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35791 } }
I20260923 22:21:07.498943 22131 raft_consensus.cc:388] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:07.498965 22131 raft_consensus.cc:482] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:07.499027 22131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:07.499533 22131 raft_consensus.cc:504] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85988eb857c4869a60f5e64f0ff1d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35791 } }
I20260923 22:21:07.499615 22131 leader_election.cc:302] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c85988eb857c4869a60f5e64f0ff1d58; no voters:
I20260923 22:21:07.499783 22131 leader_election.cc:288] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:07.499809 22136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:07.499922 22136 raft_consensus.cc:686] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [term 1 LEADER]: Becoming Leader. State: Replica: c85988eb857c4869a60f5e64f0ff1d58, State: Running, Role: LEADER
I20260923 22:21:07.500033 22131 sys_catalog.cc:564] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:07.500056 22136 consensus_queue.cc:237] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85988eb857c4869a60f5e64f0ff1d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35791 } }
I20260923 22:21:07.500391 22136 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c85988eb857c4869a60f5e64f0ff1d58. Latest consensus state: current_term: 1 leader_uuid: "c85988eb857c4869a60f5e64f0ff1d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85988eb857c4869a60f5e64f0ff1d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35791 } } }
I20260923 22:21:07.500401 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c85988eb857c4869a60f5e64f0ff1d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85988eb857c4869a60f5e64f0ff1d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35791 } } }
I20260923 22:21:07.500523 22136 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:07.500732 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:07.500874 22144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:07.501341 22144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:07.503427 22144 catalog_manager.cc:1347] Generated new cluster ID: c855f2985f784db683d6dbddb4290e6a
I20260923 22:21:07.503480 22144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:21:07 dist-test-slave-dmw7 krb5kdc[22050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 22:21:07.508558 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:07.497029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59707 (local address 127.0.0.1:35791)
0923 22:21:07.497187 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:07.497198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:07.497232 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:07.497912 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:07.497919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:07.497966 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:07.498139 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:07.498833 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:07.500253 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:07.501328 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:07.501957 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:07.502371 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:07.504809 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:07.504812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:07.504813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:07.504827 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:07.506815 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:07.507328 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:07.507333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:07.507333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:07.507443 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:07.507635 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:07.507636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:07.507637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:07.507951 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:07.508109 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:07.508217 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 22:21:07.508355 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260923 22:21:07.509478 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:07.517940 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:07.510925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43343 (local address 127.0.0.1:35791)
0923 22:21:07.510962 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:07.510963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:07.511108 (+ 145us) server_negotiation.cc:407] Connection header received
0923 22:21:07.511170 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:07.511171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:07.511180 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:07.511234 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:07.511852 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:07.512572 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:07.513567 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:07.513872 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:07.513924 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:07.515302 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:07.515303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:07.515303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:07.515310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:07.516823 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:07.517202 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:07.517203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:07.517203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:07.517255 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:07.517442 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:07.517443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:07.517443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:07.517557 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:07.517641 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:07.517771 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 22:21:07.517845 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:07.519821 22144 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:07.520288 22144 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:07.526417 22144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c85988eb857c4869a60f5e64f0ff1d58: Generated new TSK 0
I20260923 22:21:07.526597 22144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:07.530192 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22063
2026-09-23T22:21:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:07 dist-test-slave-dmw7 krb5kdc[22162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:07 dist-test-slave-dmw7 krb5kdc[22162](info): set up 2 sockets
Sep 23 22:21:07 dist-test-slave-dmw7 krb5kdc[22162](info): commencing operation
krb5kdc: starting...
W20260923 22:21:08.566668 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:08 dist-test-slave-dmw7 krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:08.681358 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:08.761379 22175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:08.761606 22175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:08.761644 22175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:08.762835 22175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:08.762918 22175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:08.762984 22175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:08.763015 22175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:08.763041 22175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:08.764460 22175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:08.764724 22175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:08.764941 22175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:21:08.769588 22175 server_base.cc:1034] running on GCE node
W20260923 22:21:08.769662 22183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:08.769590 22180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:08.769773 22181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:08.770028 22175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:08.770328 22175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:08.771476 22175 hybrid_clock.cc:648] HybridClock initialized: now 1790202068771457 us; error 30 us; skew 500 ppm
Sep 23 22:21:08 dist-test-slave-dmw7 krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:08.773772 22175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:08.774173 22175 webserver.cc:460] Webserver started at http://127.0.0.1:46493/ using document root <none> and password file <none>
I20260923 22:21:08.774377 22175 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:08.774437 22175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:08.774572 22175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:08.775583 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "8ff035f02c564470a23dbc77fd6d3bc0"
format_stamp: "Formatted at 2026-09-23 22:21:08 on dist-test-slave-dmw7"
server_key: "708a4d5ecde8676aecf7ef37c0b6f22b"
server_key_iv: "061693c7fa2ff0462d307a870dd80048"
server_key_version: "encryptionkey@0"
I20260923 22:21:08.775918 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "8ff035f02c564470a23dbc77fd6d3bc0"
format_stamp: "Formatted at 2026-09-23 22:21:08 on dist-test-slave-dmw7"
server_key: "708a4d5ecde8676aecf7ef37c0b6f22b"
server_key_iv: "061693c7fa2ff0462d307a870dd80048"
server_key_version: "encryptionkey@0"
I20260923 22:21:08.777133 22175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:08.777761 22190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:08.777906 22175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:08.778000 22175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "8ff035f02c564470a23dbc77fd6d3bc0"
format_stamp: "Formatted at 2026-09-23 22:21:08 on dist-test-slave-dmw7"
server_key: "708a4d5ecde8676aecf7ef37c0b6f22b"
server_key_iv: "061693c7fa2ff0462d307a870dd80048"
server_key_version: "encryptionkey@0"
I20260923 22:21:08.778080 22175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:08.799525 22175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:08.801842 22175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:08.802008 22175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:08.805898 22175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39605
I20260923 22:21:08.805931 22242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39605 every 8 connection(s)
I20260923 22:21:08.806267 22175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:08.807202 22243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:08.809440 22243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0: Bootstrap starting.
I20260923 22:21:08.809995 22243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:08.810274 22243 log.cc:824] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:08.810984 22243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0: No bootstrap required, opened a new log
I20260923 22:21:08.812593 22243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff035f02c564470a23dbc77fd6d3bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39605 } }
I20260923 22:21:08.812747 22243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:08.812793 22243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff035f02c564470a23dbc77fd6d3bc0, State: Initialized, Role: FOLLOWER
I20260923 22:21:08.812947 22243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff035f02c564470a23dbc77fd6d3bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39605 } }
I20260923 22:21:08.813023 22243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:08.813067 22243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:08.813117 22243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:08.813608 22243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff035f02c564470a23dbc77fd6d3bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39605 } }
I20260923 22:21:08.813724 22243 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff035f02c564470a23dbc77fd6d3bc0; no voters:
I20260923 22:21:08.813899 22243 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:08.813936 22246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:08.814095 22246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 8ff035f02c564470a23dbc77fd6d3bc0, State: Running, Role: LEADER
I20260923 22:21:08.814152 22243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:08.814227 22246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff035f02c564470a23dbc77fd6d3bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39605 } }
I20260923 22:21:08.814571 22248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff035f02c564470a23dbc77fd6d3bc0. Latest consensus state: current_term: 1 leader_uuid: "8ff035f02c564470a23dbc77fd6d3bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff035f02c564470a23dbc77fd6d3bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39605 } } }
I20260923 22:21:08.814697 22247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ff035f02c564470a23dbc77fd6d3bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff035f02c564470a23dbc77fd6d3bc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39605 } } }
I20260923 22:21:08.814736 22248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:08.814778 22247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:08.815229 22255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:08.815768 22255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:08.816110 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22175
I20260923 22:21:08.816224 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:08.817162 1541 external_mini_cluster.cc:1428] Setting key 5aa06774e7c24d40c6ddc51dea9cd801
I20260923 22:21:08.818377 22255 catalog_manager.cc:1347] Generated new cluster ID: 0906c11c070f4202a87654097506fd78
I20260923 22:21:08.818437 22255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:08.823962 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:08.817625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59103 (local address 127.0.0.1:39605)
0923 22:21:08.818744 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:08.818753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:08.818786 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:21:08.818898 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:08.818905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:08.818971 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:08.819120 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:08.819319 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:08.820743 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:08.821764 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:08.822400 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:08.822510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:08.823285 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:08.823289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:08.823289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:08.823495 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:08.823534 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:08.823549 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:08.823620 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:08.823691 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 22:21:08.823754 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":155,"threads_started":1}
I20260923 22:21:08.824857 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:08.834059 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:08.826944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50657 (local address 127.0.0.1:39605)
0923 22:21:08.826976 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:08.826978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:08.830826 (+ 3848us) server_negotiation.cc:407] Connection header received
0923 22:21:08.830888 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:08.830889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:08.830900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:08.830953 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:08.831079 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:08.831784 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:08.832789 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:08.833087 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:08.833143 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:08.833815 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:08.833817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:08.833817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:08.833823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:08.833837 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:08.833841 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:08.833892 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:08.833960 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 22:21:08.833983 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:08.835645 22255 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:08.836321 22255 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:08.841411 22255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff035f02c564470a23dbc77fd6d3bc0: Generated new TSK 0
I20260923 22:21:08.841568 22255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:08.844717 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22175
2026-09-23T22:21:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:08 dist-test-slave-dmw7 krb5kdc[22274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:08 dist-test-slave-dmw7 krb5kdc[22274](info): set up 2 sockets
Sep 23 22:21:08 dist-test-slave-dmw7 krb5kdc[22274](info): commencing operation
krb5kdc: starting...
W20260923 22:21:09.882728 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:09 dist-test-slave-dmw7 krb5kdc[22274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:09.997109 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:10.076333 22287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:10.076568 22287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:10.076606 22287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:10.077747 22287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:10.077807 22287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:10.077833 22287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:10.077862 22287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:10.077886 22287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:10.079419 22287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:10.079680 22287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:10.079890 22287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:10.084448 22292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:10.084556 22293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:10.084611 22287 server_base.cc:1034] running on GCE node
W20260923 22:21:10.084743 22295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:10.085033 22287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:10.085320 22287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:10.086478 22287 hybrid_clock.cc:648] HybridClock initialized: now 1790202070086467 us; error 35 us; skew 500 ppm
Sep 23 22:21:10 dist-test-slave-dmw7 krb5kdc[22274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:10.094710 22287 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:10.095155 22287 webserver.cc:460] Webserver started at http://127.0.0.1:44919/ using document root <none> and password file <none>
I20260923 22:21:10.095386 22287 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:10.095448 22287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:10.095594 22287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:10.096598 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "9bed5b1748c34690b955869dab8046a3"
format_stamp: "Formatted at 2026-09-23 22:21:10 on dist-test-slave-dmw7"
server_key: "c10f4409259a776c177f8c1d112c115f"
server_key_iv: "2adc4e37edc61cb62756356579b5a094"
server_key_version: "encryptionkey@0"
I20260923 22:21:10.096949 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "9bed5b1748c34690b955869dab8046a3"
format_stamp: "Formatted at 2026-09-23 22:21:10 on dist-test-slave-dmw7"
server_key: "c10f4409259a776c177f8c1d112c115f"
server_key_iv: "2adc4e37edc61cb62756356579b5a094"
server_key_version: "encryptionkey@0"
I20260923 22:21:10.098204 22287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:21:10.098891 22302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:10.099048 22287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:10.099138 22287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "9bed5b1748c34690b955869dab8046a3"
format_stamp: "Formatted at 2026-09-23 22:21:10 on dist-test-slave-dmw7"
server_key: "c10f4409259a776c177f8c1d112c115f"
server_key_iv: "2adc4e37edc61cb62756356579b5a094"
server_key_version: "encryptionkey@0"
I20260923 22:21:10.099218 22287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:10.102280 22287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:10.104389 22287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:10.104530 22287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:10.108338 22287 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41799
I20260923 22:21:10.108367 22354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41799 every 8 connection(s)
I20260923 22:21:10.108676 22287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:10.109549 22355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:10.111064 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22287
I20260923 22:21:10.111168 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:10.111344 1541 external_mini_cluster.cc:1428] Setting key eb256e230fb05d463d55a6373b063b75
I20260923 22:21:10.112030 22355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3: Bootstrap starting.
I20260923 22:21:10.112699 22355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:10.113001 22355 log.cc:824] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:10.113622 22355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3: No bootstrap required, opened a new log
I20260923 22:21:10.114147 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:10.111807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51627 (local address 127.0.0.1:41799)
0923 22:21:10.112000 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:10.112016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:10.112046 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:21:10.112640 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:10.112648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:10.112681 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:10.112828 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:10.113734 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:10.113737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:10.113738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:10.113742 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:10.113781 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:10.113798 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:10.113881 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:10.113949 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 22:21:10.113989 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260923 22:21:10.115038 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:10.115917 22355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed5b1748c34690b955869dab8046a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41799 } }
I20260923 22:21:10.116084 22355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:10.116133 22355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bed5b1748c34690b955869dab8046a3, State: Initialized, Role: FOLLOWER
I20260923 22:21:10.116247 22355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed5b1748c34690b955869dab8046a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41799 } }
I20260923 22:21:10.116341 22355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:10.116407 22355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:10.116487 22355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:10.117208 22355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed5b1748c34690b955869dab8046a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41799 } }
I20260923 22:21:10.117312 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:10.116327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60479 (local address 127.0.0.1:41799)
0923 22:21:10.116361 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:10.116363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:10.116490 (+ 127us) server_negotiation.cc:407] Connection header received
0923 22:21:10.116543 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:10.116544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:10.116548 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:10.116614 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:10.117105 (+ 491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:10.117106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:10.117106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:10.117110 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:10.117125 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:10.117129 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:10.117211 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:10.117252 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 22:21:10.117264 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:10.117319 22355 leader_election.cc:302] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bed5b1748c34690b955869dab8046a3; no voters:
I20260923 22:21:10.117527 22355 leader_election.cc:288] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:10.117556 22365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:10.117731 22365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [term 1 LEADER]: Becoming Leader. State: Replica: 9bed5b1748c34690b955869dab8046a3, State: Running, Role: LEADER
I20260923 22:21:10.117748 22355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:10.117841 22365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed5b1748c34690b955869dab8046a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41799 } }
I20260923 22:21:10.118161 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bed5b1748c34690b955869dab8046a3. Latest consensus state: current_term: 1 leader_uuid: "9bed5b1748c34690b955869dab8046a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed5b1748c34690b955869dab8046a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41799 } } }
I20260923 22:21:10.118166 22366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bed5b1748c34690b955869dab8046a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bed5b1748c34690b955869dab8046a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41799 } } }
I20260923 22:21:10.118299 22366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:10.118486 22372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:10.118494 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:10.119000 22372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:10.123567 22372 catalog_manager.cc:1347] Generated new cluster ID: c7b9730e5eb24d268aa70fa7a537c8d0
I20260923 22:21:10.123621 22372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:10.138312 22372 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:10.138460 22372 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:10.149642 22372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bed5b1748c34690b955869dab8046a3: Generated new TSK 0
I20260923 22:21:10.149835 22372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:10.152382 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22287
2026-09-23T22:21:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:10 dist-test-slave-dmw7 krb5kdc[22386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:10 dist-test-slave-dmw7 krb5kdc[22386](info): set up 2 sockets
Sep 23 22:21:10 dist-test-slave-dmw7 krb5kdc[22386](info): commencing operation
krb5kdc: starting...
W20260923 22:21:11.182796 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:11 dist-test-slave-dmw7 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:11.297092 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:11.376384 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:11.376575 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:11.376598 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:11.377753 22399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:11.377794 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:11.377808 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:11.377821 22399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:11.377876 22399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:11.379304 22399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:11.379554 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:11.379777 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:11.384536 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:11.384613 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:11.384511 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:11.384713 22399 server_base.cc:1034] running on GCE node
I20260923 22:21:11.385016 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:11.385285 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:11.386430 22399 hybrid_clock.cc:648] HybridClock initialized: now 1790202071386412 us; error 35 us; skew 500 ppm
Sep 23 22:21:11 dist-test-slave-dmw7 krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:11.388881 22399 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:11.389292 22399 webserver.cc:460] Webserver started at http://127.0.0.1:35281/ using document root <none> and password file <none>
I20260923 22:21:11.389506 22399 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:11.389568 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:11.389693 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:11.390842 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "00954209201944749f5af1d336a27e42"
format_stamp: "Formatted at 2026-09-23 22:21:11 on dist-test-slave-dmw7"
server_key: "dc9a82deb85afe2c7d532b128c3367ba"
server_key_iv: "b4c9d2f44b40a9fd64eeb050b7e0168c"
server_key_version: "encryptionkey@0"
I20260923 22:21:11.391218 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "00954209201944749f5af1d336a27e42"
format_stamp: "Formatted at 2026-09-23 22:21:11 on dist-test-slave-dmw7"
server_key: "dc9a82deb85afe2c7d532b128c3367ba"
server_key_iv: "b4c9d2f44b40a9fd64eeb050b7e0168c"
server_key_version: "encryptionkey@0"
I20260923 22:21:11.392539 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:11.393369 22414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:11.393574 22399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:11.393661 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "00954209201944749f5af1d336a27e42"
format_stamp: "Formatted at 2026-09-23 22:21:11 on dist-test-slave-dmw7"
server_key: "dc9a82deb85afe2c7d532b128c3367ba"
server_key_iv: "b4c9d2f44b40a9fd64eeb050b7e0168c"
server_key_version: "encryptionkey@0"
I20260923 22:21:11.393735 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:11.404863 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:11.406800 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:11.406961 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:11.411079 22399 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40313
I20260923 22:21:11.411098 22466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40313 every 8 connection(s)
I20260923 22:21:11.411460 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:11.412401 22467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:11.414547 22467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42: Bootstrap starting.
I20260923 22:21:11.415138 22467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:11.415431 22467 log.cc:824] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:11.416119 22467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42: No bootstrap required, opened a new log
I20260923 22:21:11.417649 22467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00954209201944749f5af1d336a27e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40313 } }
I20260923 22:21:11.417785 22467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:11.417809 22467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00954209201944749f5af1d336a27e42, State: Initialized, Role: FOLLOWER
I20260923 22:21:11.417958 22467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00954209201944749f5af1d336a27e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40313 } }
I20260923 22:21:11.418063 22467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:11.418108 22467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:11.418164 22467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:11.418720 22467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00954209201944749f5af1d336a27e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40313 } }
I20260923 22:21:11.418834 22467 leader_election.cc:302] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00954209201944749f5af1d336a27e42; no voters:
I20260923 22:21:11.419005 22467 leader_election.cc:288] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:11.419097 22470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:11.419281 22470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [term 1 LEADER]: Becoming Leader. State: Replica: 00954209201944749f5af1d336a27e42, State: Running, Role: LEADER
I20260923 22:21:11.419342 22467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:11.419404 22470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00954209201944749f5af1d336a27e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40313 } }
I20260923 22:21:11.419803 22471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00954209201944749f5af1d336a27e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00954209201944749f5af1d336a27e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40313 } } }
I20260923 22:21:11.419812 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00954209201944749f5af1d336a27e42. Latest consensus state: current_term: 1 leader_uuid: "00954209201944749f5af1d336a27e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00954209201944749f5af1d336a27e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40313 } } }
I20260923 22:21:11.419909 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:11.420060 22471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:11.420105 22474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:11.420644 22474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:11.421327 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22399
I20260923 22:21:11.421447 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:11.421617 1541 external_mini_cluster.cc:1428] Setting key f6b0a8f49270d40657790138a6194d90
I20260923 22:21:11.422323 22474 catalog_manager.cc:1347] Generated new cluster ID: c3881e8caf5740648022799432a56fe8
I20260923 22:21:11.422386 22474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:11.426596 22474 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:11.427175 22474 catalog_manager.cc:1504] Loading token signing keys...
Sep 23 22:21:11 dist-test-slave-dmw7 krb5kdc[22386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 22:21:11.433604 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:11.422135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:60597 (local address 127.0.0.1:40313)
0923 22:21:11.422654 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:11.422664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:11.422695 (+ 31us) server_negotiation.cc:407] Connection header received
0923 22:21:11.422808 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:11.422815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:11.422874 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:11.423013 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:11.423713 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:11.425159 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:11.426201 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:11.426941 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:11.427071 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:11.429929 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:11.429934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:11.429934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:11.429945 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:11.431922 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:11.432383 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:11.432386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:11.432387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:11.432485 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:11.432724 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:11.432725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:11.432725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:11.432980 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:11.433094 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:11.433282 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 22:21:11.433378 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":69,"threads_started":1}
I20260923 22:21:11.434607 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:11.435040 22474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00954209201944749f5af1d336a27e42: Generated new TSK 0
I20260923 22:21:11.435199 22474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:11.441900 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:11.436024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:39427 (local address 127.0.0.1:40313)
0923 22:21:11.436083 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:11.436085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:11.436259 (+ 174us) server_negotiation.cc:407] Connection header received
0923 22:21:11.436333 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:11.436333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:11.436344 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:11.436393 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:11.436900 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:11.437406 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:11.438138 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:11.438377 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:11.438462 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:11.439584 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:11.439585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:11.439586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:11.439592 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:11.440824 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:11.441156 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:11.441157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:11.441157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:11.441222 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:11.441431 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:11.441432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:11.441433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:11.441537 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:11.441640 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:11.441754 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 22:21:11.441812 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:21:11.443650 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22399
W20260923 22:21:11.447517 22384 connection.cc:535] client connection to 127.0.0.1:40313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40313): Connection reset by peer (error 104)
2026-09-23T22:21:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:11 dist-test-slave-dmw7 krb5kdc[22498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:11 dist-test-slave-dmw7 krb5kdc[22498](info): set up 2 sockets
Sep 23 22:21:11 dist-test-slave-dmw7 krb5kdc[22498](info): commencing operation
krb5kdc: starting...
W20260923 22:21:12.474787 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:12 dist-test-slave-dmw7 krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:12.588709 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:12.669601 22511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:12.669838 22511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:12.669879 22511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:12.671069 22511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:12.671139 22511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:12.671167 22511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:12.671205 22511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:12.671255 22511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:12.672727 22511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:12.673012 22511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:12.673243 22511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:12.678244 22516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:12.678375 22511 server_base.cc:1034] running on GCE node
W20260923 22:21:12.678231 22517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:12.678231 22519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:12.678766 22511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:12.679062 22511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:12.680223 22511 hybrid_clock.cc:648] HybridClock initialized: now 1790202072680214 us; error 29 us; skew 500 ppm
Sep 23 22:21:12 dist-test-slave-dmw7 krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:12.682576 22511 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:12.683032 22511 webserver.cc:460] Webserver started at http://127.0.0.1:34927/ using document root <none> and password file <none>
I20260923 22:21:12.683276 22511 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:12.683346 22511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:12.683488 22511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:12.684497 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "acc3a5dab77b41a1bf6598781fd6f26c"
format_stamp: "Formatted at 2026-09-23 22:21:12 on dist-test-slave-dmw7"
server_key: "a69c297821b329c9072fdfab31a8ec24"
server_key_iv: "6fd3f809ef88f20863acdd3ee75d05e4"
server_key_version: "encryptionkey@0"
I20260923 22:21:12.684854 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "acc3a5dab77b41a1bf6598781fd6f26c"
format_stamp: "Formatted at 2026-09-23 22:21:12 on dist-test-slave-dmw7"
server_key: "a69c297821b329c9072fdfab31a8ec24"
server_key_iv: "6fd3f809ef88f20863acdd3ee75d05e4"
server_key_version: "encryptionkey@0"
I20260923 22:21:12.686134 22511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:12.686884 22526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:12.687067 22511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:12.687166 22511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "acc3a5dab77b41a1bf6598781fd6f26c"
format_stamp: "Formatted at 2026-09-23 22:21:12 on dist-test-slave-dmw7"
server_key: "a69c297821b329c9072fdfab31a8ec24"
server_key_iv: "6fd3f809ef88f20863acdd3ee75d05e4"
server_key_version: "encryptionkey@0"
I20260923 22:21:12.687240 22511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:12.697836 22511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:12.699558 22511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:12.699714 22511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:12.703456 22511 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38705
I20260923 22:21:12.703524 22578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38705 every 8 connection(s)
I20260923 22:21:12.703853 22511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:12.704819 22579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:12.707134 22579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c: Bootstrap starting.
I20260923 22:21:12.707674 22579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:12.707960 22579 log.cc:824] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:12.708619 22579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c: No bootstrap required, opened a new log
I20260923 22:21:12.710116 22579 raft_consensus.cc:348] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a5dab77b41a1bf6598781fd6f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } }
I20260923 22:21:12.710249 22579 raft_consensus.cc:374] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:12.710302 22579 raft_consensus.cc:729] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc3a5dab77b41a1bf6598781fd6f26c, State: Initialized, Role: FOLLOWER
I20260923 22:21:12.710449 22579 consensus_queue.cc:260] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a5dab77b41a1bf6598781fd6f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } }
I20260923 22:21:12.710529 22579 raft_consensus.cc:388] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:12.710593 22579 raft_consensus.cc:482] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:12.710668 22579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:12.711158 22579 raft_consensus.cc:504] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a5dab77b41a1bf6598781fd6f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } }
I20260923 22:21:12.711267 22579 leader_election.cc:302] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acc3a5dab77b41a1bf6598781fd6f26c; no voters:
I20260923 22:21:12.711440 22579 leader_election.cc:288] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:12.711488 22582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:12.711634 22582 raft_consensus.cc:686] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [term 1 LEADER]: Becoming Leader. State: Replica: acc3a5dab77b41a1bf6598781fd6f26c, State: Running, Role: LEADER
I20260923 22:21:12.711690 22579 sys_catalog.cc:564] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:12.711762 22582 consensus_queue.cc:237] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a5dab77b41a1bf6598781fd6f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } }
I20260923 22:21:12.712365 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acc3a5dab77b41a1bf6598781fd6f26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a5dab77b41a1bf6598781fd6f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } } }
I20260923 22:21:12.712383 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [sys.catalog]: SysCatalogTable state changed. Reason: New leader acc3a5dab77b41a1bf6598781fd6f26c. Latest consensus state: current_term: 1 leader_uuid: "acc3a5dab77b41a1bf6598781fd6f26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a5dab77b41a1bf6598781fd6f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38705 } } }
I20260923 22:21:12.712476 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:12.712464 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:12.712705 22591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:12.713176 22591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:12.713471 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22511
I20260923 22:21:12.713590 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:12.713775 1541 external_mini_cluster.cc:1428] Setting key 8cb603520b9903e32d05f5811b82c60e
I20260923 22:21:12.715413 22591 catalog_manager.cc:1347] Generated new cluster ID: 86df7a85ab1b45b0959e37963fdac23d
I20260923 22:21:12.715469 22591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:12.720695 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:12.714227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:52793 (local address 127.0.0.1:38705)
0923 22:21:12.714401 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:12.714410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:12.714888 (+ 478us) server_negotiation.cc:407] Connection header received
0923 22:21:12.715016 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:12.715023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:12.715097 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:12.715242 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:12.715460 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:12.716862 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:12.717929 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:12.718972 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:12.719803 (+ 831us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:12.719939 (+ 136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:12.719942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:12.719942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:12.720129 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:12.720162 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:12.720175 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:12.720428 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:12.720466 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 22:21:12.720490 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260923 22:21:12.721769 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:12.726471 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:12.723208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:34535 (local address 127.0.0.1:38705)
0923 22:21:12.723239 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:12.723241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:12.723307 (+ 66us) server_negotiation.cc:407] Connection header received
0923 22:21:12.723374 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:12.723375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:12.723383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:12.723436 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:12.723561 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:12.724236 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:12.725246 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:12.725542 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:12.725598 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:12.726240 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:12.726241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:12.726241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:12.726245 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:12.726258 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:12.726262 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:12.726316 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:12.726397 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 22:21:12.726408 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 22:21:12.731770 22591 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:12.732234 22591 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:12.735704 22591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acc3a5dab77b41a1bf6598781fd6f26c: Generated new TSK 0
I20260923 22:21:12.735875 22591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:12.739033 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22511
W20260923 22:21:12.743379 22496 connection.cc:535] client connection to 127.0.0.1:38705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38705): Connection reset by peer (error 104)
2026-09-23T22:21:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:12 dist-test-slave-dmw7 krb5kdc[22610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:12 dist-test-slave-dmw7 krb5kdc[22610](info): set up 2 sockets
Sep 23 22:21:12 dist-test-slave-dmw7 krb5kdc[22610](info): commencing operation
krb5kdc: starting...
W20260923 22:21:13.767518 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:13 dist-test-slave-dmw7 krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:13.896713 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:13.976380 22623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:13.976634 22623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:13.976672 22623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:13.977847 22623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:13.977923 22623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:13.977947 22623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:13.977977 22623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:13.978017 22623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:13.979553 22623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:13.979822 22623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:13.980059 22623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:13.985352 22629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:13.985327 22631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:13.985402 22628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:13.985526 22623 server_base.cc:1034] running on GCE node
I20260923 22:21:13.985837 22623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:13.986130 22623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:13.987318 22623 hybrid_clock.cc:648] HybridClock initialized: now 1790202073987305 us; error 27 us; skew 500 ppm
Sep 23 22:21:13 dist-test-slave-dmw7 krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:13.989877 22623 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:13.990278 22623 webserver.cc:460] Webserver started at http://127.0.0.1:36369/ using document root <none> and password file <none>
I20260923 22:21:13.990538 22623 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:13.990589 22623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:13.990793 22623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:13.991884 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "3ab02164868f4982957e4a3df2518362"
format_stamp: "Formatted at 2026-09-23 22:21:13 on dist-test-slave-dmw7"
server_key: "5a13fef10cc3d18f7944631f7e730d2e"
server_key_iv: "ce0668d0ef2ae984cad7e0bd067490b5"
server_key_version: "encryptionkey@0"
I20260923 22:21:13.992257 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "3ab02164868f4982957e4a3df2518362"
format_stamp: "Formatted at 2026-09-23 22:21:13 on dist-test-slave-dmw7"
server_key: "5a13fef10cc3d18f7944631f7e730d2e"
server_key_iv: "ce0668d0ef2ae984cad7e0bd067490b5"
server_key_version: "encryptionkey@0"
I20260923 22:21:13.993613 22623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:13.994356 22638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:13.994540 22623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:13.994634 22623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "3ab02164868f4982957e4a3df2518362"
format_stamp: "Formatted at 2026-09-23 22:21:13 on dist-test-slave-dmw7"
server_key: "5a13fef10cc3d18f7944631f7e730d2e"
server_key_iv: "ce0668d0ef2ae984cad7e0bd067490b5"
server_key_version: "encryptionkey@0"
I20260923 22:21:13.994732 22623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:13.998055 22623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:13.999969 22623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:14.000133 22623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:14.004348 22623 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35263
I20260923 22:21:14.004379 22690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35263 every 8 connection(s)
I20260923 22:21:14.004730 22623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:14.005677 22691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:14.007926 22691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362: Bootstrap starting.
I20260923 22:21:14.008492 22691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:14.008795 22691 log.cc:824] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:14.009505 22691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362: No bootstrap required, opened a new log
I20260923 22:21:14.011256 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22623
I20260923 22:21:14.011377 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:14.011257 22691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab02164868f4982957e4a3df2518362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35263 } }
I20260923 22:21:14.011430 22691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:14.011477 22691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab02164868f4982957e4a3df2518362, State: Initialized, Role: FOLLOWER
I20260923 22:21:14.011579 1541 external_mini_cluster.cc:1428] Setting key 7039d4db26e9fba5536e493554592704
I20260923 22:21:14.011641 22691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab02164868f4982957e4a3df2518362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35263 } }
I20260923 22:21:14.011725 22691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:14.011771 22691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:14.011818 22691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:14.012925 22691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab02164868f4982957e4a3df2518362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35263 } }
I20260923 22:21:14.013042 22691 leader_election.cc:302] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ab02164868f4982957e4a3df2518362; no voters:
I20260923 22:21:14.013228 22691 leader_election.cc:288] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:14.013247 22696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:14.013409 22696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab02164868f4982957e4a3df2518362, State: Running, Role: LEADER
I20260923 22:21:14.013558 22691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:14.013538 22696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab02164868f4982957e4a3df2518362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35263 } }
I20260923 22:21:14.013952 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:14.012138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:40819 (local address 127.0.0.1:35263)
0923 22:21:14.012321 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:14.012332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:14.012365 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:21:14.012465 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:14.012471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:14.012493 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:14.012606 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:14.013333 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:14.013337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:14.013338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:14.013342 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:14.013397 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:14.013413 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:14.013671 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:14.013702 (+ 31us) server_negotiation.cc:299] Negotiation successful
0923 22:21:14.013741 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260923 22:21:14.013979 22697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ab02164868f4982957e4a3df2518362. Latest consensus state: current_term: 1 leader_uuid: "3ab02164868f4982957e4a3df2518362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab02164868f4982957e4a3df2518362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35263 } } }
I20260923 22:21:14.014176 22697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:14.014557 22698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ab02164868f4982957e4a3df2518362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab02164868f4982957e4a3df2518362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35263 } } }
I20260923 22:21:14.014681 22698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:14.015027 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:14.014873 22705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:14.015589 22705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:14.017268 22705 catalog_manager.cc:1347] Generated new cluster ID: 415454a989c8401ca50dd52731a5e083
I20260923 22:21:14.017313 22705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:14.019099 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:14.016548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:47345 (local address 127.0.0.1:35263)
0923 22:21:14.016589 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:14.016592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:14.016731 (+ 139us) server_negotiation.cc:407] Connection header received
0923 22:21:14.016788 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:14.016789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:14.016795 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:14.016850 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:14.018862 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:14.018864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:14.018864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:14.018868 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:14.018885 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:14.018889 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:14.018957 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:14.019014 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 22:21:14.019035 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:14.026444 22705 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:14.026605 22705 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:14.037699 22705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ab02164868f4982957e4a3df2518362: Generated new TSK 0
I20260923 22:21:14.037909 22705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:14.045969 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22623
2026-09-23T22:21:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:14 dist-test-slave-dmw7 krb5kdc[22722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:14 dist-test-slave-dmw7 krb5kdc[22722](info): set up 2 sockets
Sep 23 22:21:14 dist-test-slave-dmw7 krb5kdc[22722](info): commencing operation
krb5kdc: starting...
W20260923 22:21:15.074815 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:15 dist-test-slave-dmw7 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:15.189900 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:15.268689 22735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:15.268934 22735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:15.268971 22735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:15.270121 22735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:15.270192 22735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:15.270223 22735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:15.270253 22735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:15.270284 22735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:15.271783 22735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:15.272063 22735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:15.272294 22735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:15.277048 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:15.277153 22735 server_base.cc:1034] running on GCE node
W20260923 22:21:15.277091 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:15.277100 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:15.277559 22735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:15.277868 22735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:15.279004 22735 hybrid_clock.cc:648] HybridClock initialized: now 1790202075278980 us; error 36 us; skew 500 ppm
Sep 23 22:21:15 dist-test-slave-dmw7 krb5kdc[22722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:15.286358 22735 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:15.286856 22735 webserver.cc:460] Webserver started at http://127.0.0.1:35493/ using document root <none> and password file <none>
I20260923 22:21:15.287086 22735 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:15.287151 22735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:15.287298 22735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:15.288340 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "c7066f8f673b42cea1ca0f3bb079c755"
format_stamp: "Formatted at 2026-09-23 22:21:15 on dist-test-slave-dmw7"
server_key: "09fd7375737072aa03029645d314289a"
server_key_iv: "799a41e100e8decec401e5c1abdc5bd6"
server_key_version: "encryptionkey@0"
I20260923 22:21:15.288699 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "c7066f8f673b42cea1ca0f3bb079c755"
format_stamp: "Formatted at 2026-09-23 22:21:15 on dist-test-slave-dmw7"
server_key: "09fd7375737072aa03029645d314289a"
server_key_iv: "799a41e100e8decec401e5c1abdc5bd6"
server_key_version: "encryptionkey@0"
I20260923 22:21:15.289986 22735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:15.290704 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:15.290901 22735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:15.290998 22735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "c7066f8f673b42cea1ca0f3bb079c755"
format_stamp: "Formatted at 2026-09-23 22:21:15 on dist-test-slave-dmw7"
server_key: "09fd7375737072aa03029645d314289a"
server_key_iv: "799a41e100e8decec401e5c1abdc5bd6"
server_key_version: "encryptionkey@0"
I20260923 22:21:15.291082 22735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:15.312222 22735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:15.314667 22735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:15.314833 22735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:15.318580 22735 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40327
I20260923 22:21:15.318686 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40327 every 8 connection(s)
I20260923 22:21:15.319010 22735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:15.319926 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:15.322052 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755: Bootstrap starting.
I20260923 22:21:15.322576 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:15.322883 22803 log.cc:824] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:15.323567 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755: No bootstrap required, opened a new log
I20260923 22:21:15.324877 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22735
I20260923 22:21:15.325008 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:15.325192 1541 external_mini_cluster.cc:1428] Setting key 23d7595f595a58802928bc6ff93e02b0
I20260923 22:21:15.325388 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7066f8f673b42cea1ca0f3bb079c755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40327 } }
I20260923 22:21:15.325553 22803 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:15.325819 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7066f8f673b42cea1ca0f3bb079c755, State: Initialized, Role: FOLLOWER
I20260923 22:21:15.325968 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7066f8f673b42cea1ca0f3bb079c755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40327 } }
I20260923 22:21:15.326054 22803 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:15.326098 22803 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:15.326145 22803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:15.327033 22803 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7066f8f673b42cea1ca0f3bb079c755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40327 } }
I20260923 22:21:15.327147 22803 leader_election.cc:302] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7066f8f673b42cea1ca0f3bb079c755; no voters:
I20260923 22:21:15.327342 22803 leader_election.cc:288] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:15.327376 22808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:15.327473 22808 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [term 1 LEADER]: Becoming Leader. State: Replica: c7066f8f673b42cea1ca0f3bb079c755, State: Running, Role: LEADER
I20260923 22:21:15.327593 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:15.327600 22808 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7066f8f673b42cea1ca0f3bb079c755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40327 } }
I20260923 22:21:15.328004 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7066f8f673b42cea1ca0f3bb079c755. Latest consensus state: current_term: 1 leader_uuid: "c7066f8f673b42cea1ca0f3bb079c755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7066f8f673b42cea1ca0f3bb079c755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40327 } } }
I20260923 22:21:15.328135 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:15.328317 22809 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7066f8f673b42cea1ca0f3bb079c755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7066f8f673b42cea1ca0f3bb079c755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40327 } } }
I20260923 22:21:15.328460 22809 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:15.328574 22815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:15.329303 22815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:15.331177 22815 catalog_manager.cc:1347] Generated new cluster ID: 69d7392e787643d491a3a9af954c94b7
I20260923 22:21:15.331229 22815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:21:15 dist-test-slave-dmw7 krb5kdc[22722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 22:21:15.336650 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:15.325668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:48157 (local address 127.0.0.1:40327)
0923 22:21:15.325840 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:15.325850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:15.325887 (+ 37us) server_negotiation.cc:407] Connection header received
0923 22:21:15.326030 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:15.326036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:15.326106 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:15.326242 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:15.326948 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:15.328383 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:15.329539 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:15.329980 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:15.330116 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:15.332943 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:15.332947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:15.332947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:15.332959 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:15.334918 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:15.335378 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:15.335381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:15.335381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:15.335496 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:15.335739 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:15.335739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:15.335740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:15.336023 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:15.336184 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:15.336335 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 22:21:15.336431 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260923 22:21:15.337538 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:15.342208 22815 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:15.342953 22815 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:15.345178 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:15.338864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:34905 (local address 127.0.0.1:40327)
0923 22:21:15.338902 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:15.338904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:15.339012 (+ 108us) server_negotiation.cc:407] Connection header received
0923 22:21:15.339082 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:15.339083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:15.339093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:15.339147 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:15.339641 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:15.340142 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:15.340870 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:15.341097 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:15.341173 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:15.342559 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:15.342560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:15.342561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:15.342568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:15.344028 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:15.344392 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:15.344393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:15.344394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:15.344449 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:15.344674 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:15.344674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:15.344675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:15.344785 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:15.344896 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:15.345017 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 22:21:15.345078 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:15.349097 22815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7066f8f673b42cea1ca0f3bb079c755: Generated new TSK 0
I20260923 22:21:15.349303 22815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:15.352737 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22735
W20260923 22:21:15.356787 22720 connection.cc:535] client connection to 127.0.0.1:40327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40327): Connection reset by peer (error 104)
2026-09-23T22:21:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:15 dist-test-slave-dmw7 krb5kdc[22834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:15 dist-test-slave-dmw7 krb5kdc[22834](info): set up 2 sockets
Sep 23 22:21:15 dist-test-slave-dmw7 krb5kdc[22834](info): commencing operation
krb5kdc: starting...
W20260923 22:21:16.378688 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:16 dist-test-slave-dmw7 krb5kdc[22834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:16.497239 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:16.577221 22847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:16.577421 22847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:16.577443 22847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:16.578509 22847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:16.578544 22847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:16.578558 22847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:16.578572 22847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:16.578583 22847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:16.580075 22847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:16.580279 22847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:16.580441 22847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:16.585124 22852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:16.585175 22847 server_base.cc:1034] running on GCE node
W20260923 22:21:16.585157 22853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:16.585157 22855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:16.585562 22847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:16.585860 22847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:16.586998 22847 hybrid_clock.cc:648] HybridClock initialized: now 1790202076586981 us; error 37 us; skew 500 ppm
Sep 23 22:21:16 dist-test-slave-dmw7 krb5kdc[22834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:16.589418 22847 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:16.589848 22847 webserver.cc:460] Webserver started at http://127.0.0.1:39193/ using document root <none> and password file <none>
I20260923 22:21:16.590078 22847 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:16.590147 22847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:16.590303 22847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:16.591418 22847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "fdb038a8cbf3487f8b197e54067d9f8b"
format_stamp: "Formatted at 2026-09-23 22:21:16 on dist-test-slave-dmw7"
server_key: "47f561926b57836e1121699711f83a2f"
server_key_iv: "4bd72e398bbac24b2c636f2745eec6c3"
server_key_version: "encryptionkey@0"
I20260923 22:21:16.591809 22847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "fdb038a8cbf3487f8b197e54067d9f8b"
format_stamp: "Formatted at 2026-09-23 22:21:16 on dist-test-slave-dmw7"
server_key: "47f561926b57836e1121699711f83a2f"
server_key_iv: "4bd72e398bbac24b2c636f2745eec6c3"
server_key_version: "encryptionkey@0"
I20260923 22:21:16.593145 22847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:21:16.593914 22862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:16.594106 22847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:16.594204 22847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "fdb038a8cbf3487f8b197e54067d9f8b"
format_stamp: "Formatted at 2026-09-23 22:21:16 on dist-test-slave-dmw7"
server_key: "47f561926b57836e1121699711f83a2f"
server_key_iv: "4bd72e398bbac24b2c636f2745eec6c3"
server_key_version: "encryptionkey@0"
I20260923 22:21:16.594287 22847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:16.615964 22847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:16.618350 22847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:16.618515 22847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:16.622390 22847 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36699
I20260923 22:21:16.622426 22914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36699 every 8 connection(s)
I20260923 22:21:16.622843 22847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:16.623696 22915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:16.625888 22915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b: Bootstrap starting.
I20260923 22:21:16.626395 22915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:16.626681 22915 log.cc:824] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:16.627334 22915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b: No bootstrap required, opened a new log
I20260923 22:21:16.628831 22915 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb038a8cbf3487f8b197e54067d9f8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } }
I20260923 22:21:16.628962 22915 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:16.628984 22915 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb038a8cbf3487f8b197e54067d9f8b, State: Initialized, Role: FOLLOWER
I20260923 22:21:16.629141 22915 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb038a8cbf3487f8b197e54067d9f8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } }
I20260923 22:21:16.629221 22915 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:16.629243 22915 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:16.629272 22915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:16.629735 22915 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb038a8cbf3487f8b197e54067d9f8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } }
I20260923 22:21:16.629814 22915 leader_election.cc:302] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb038a8cbf3487f8b197e54067d9f8b; no voters:
I20260923 22:21:16.629961 22915 leader_election.cc:288] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:16.629985 22918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:16.630103 22918 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [term 1 LEADER]: Becoming Leader. State: Replica: fdb038a8cbf3487f8b197e54067d9f8b, State: Running, Role: LEADER
I20260923 22:21:16.630223 22915 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:16.630251 22918 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb038a8cbf3487f8b197e54067d9f8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } }
I20260923 22:21:16.630721 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb038a8cbf3487f8b197e54067d9f8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb038a8cbf3487f8b197e54067d9f8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } } }
I20260923 22:21:16.630823 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:16.630865 22920 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb038a8cbf3487f8b197e54067d9f8b. Latest consensus state: current_term: 1 leader_uuid: "fdb038a8cbf3487f8b197e54067d9f8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb038a8cbf3487f8b197e54067d9f8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } } }
I20260923 22:21:16.630998 22920 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:16.631102 22926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:16.631654 22926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:16.632349 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22847
I20260923 22:21:16.632465 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:16.632637 1541 external_mini_cluster.cc:1428] Setting key 6ddf4bb8417da9443b0b43bd3bd21005
I20260923 22:21:16.633448 22926 catalog_manager.cc:1347] Generated new cluster ID: 9659be97a5624bc184bc3b84f3cd85be
I20260923 22:21:16.633505 22926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:16.638146 22926 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:16.638850 22926 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:16.639010 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:16.633101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:46977 (local address 127.0.0.1:36699)
0923 22:21:16.633526 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:16.633535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:16.633577 (+ 42us) server_negotiation.cc:407] Connection header received
0923 22:21:16.633698 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:16.633705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:16.633758 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:16.633882 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:16.634116 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:16.635503 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:16.636622 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:16.637210 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:16.637314 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:16.638181 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:16.638185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:16.638185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:16.638387 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:16.638425 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:16.638439 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:16.638518 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:16.638644 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 22:21:16.638761 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":76,"threads_started":1}
I20260923 22:21:16.639824 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:16.644716 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:16.641282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:51425 (local address 127.0.0.1:36699)
0923 22:21:16.641318 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:16.641320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:16.641404 (+ 84us) server_negotiation.cc:407] Connection header received
0923 22:21:16.641431 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:16.641432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:16.641452 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:16.641520 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:16.641642 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:16.642417 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:16.643417 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:16.643761 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:16.643827 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:16.644473 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:16.644475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:16.644475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:16.644479 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:16.644493 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:16.644498 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:16.644559 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:16.644632 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 22:21:16.644644 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:21:16.647311 22926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb038a8cbf3487f8b197e54067d9f8b: Generated new TSK 0
I20260923 22:21:16.647477 22926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:16.651672 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22847
W20260923 22:21:16.655889 22832 connection.cc:535] client connection to 127.0.0.1:36699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36699): Connection reset by peer (error 104)
2026-09-23T22:21:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:16 dist-test-slave-dmw7 krb5kdc[22946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:16 dist-test-slave-dmw7 krb5kdc[22946](info): set up 2 sockets
Sep 23 22:21:16 dist-test-slave-dmw7 krb5kdc[22946](info): commencing operation
krb5kdc: starting...
W20260923 22:21:17.682827 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:17 dist-test-slave-dmw7 krb5kdc[22946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:17.799798 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:17.885555 22959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:17.885774 22959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:17.885798 22959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:17.887023 22959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:17.887079 22959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:17.887094 22959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:17.887108 22959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:17.887120 22959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:17.888598 22959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:17.888798 22959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:17.888968 22959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:17.893801 22965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:17.893807 22964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:17.893807 22967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:17.894170 22959 server_base.cc:1034] running on GCE node
I20260923 22:21:17.894438 22959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:17.894789 22959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:17.895970 22959 hybrid_clock.cc:648] HybridClock initialized: now 1790202077895960 us; error 33 us; skew 500 ppm
Sep 23 22:21:17 dist-test-slave-dmw7 krb5kdc[22946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:17.898524 22959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:17.899008 22959 webserver.cc:460] Webserver started at http://127.0.0.1:35221/ using document root <none> and password file <none>
I20260923 22:21:17.899197 22959 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:17.899241 22959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:17.899330 22959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:17.900359 22959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "5bbe92692bf44e7fba2ec1f17460b159"
format_stamp: "Formatted at 2026-09-23 22:21:17 on dist-test-slave-dmw7"
server_key: "13741b17c9232d3c5aa8ed6501bbfd7f"
server_key_iv: "db3310edfecda7cb571d07d1b0379071"
server_key_version: "encryptionkey@0"
I20260923 22:21:17.900660 22959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "5bbe92692bf44e7fba2ec1f17460b159"
format_stamp: "Formatted at 2026-09-23 22:21:17 on dist-test-slave-dmw7"
server_key: "13741b17c9232d3c5aa8ed6501bbfd7f"
server_key_iv: "db3310edfecda7cb571d07d1b0379071"
server_key_version: "encryptionkey@0"
I20260923 22:21:17.901984 22959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:17.902745 22974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:17.902904 22959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:21:17.902993 22959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "5bbe92692bf44e7fba2ec1f17460b159"
format_stamp: "Formatted at 2026-09-23 22:21:17 on dist-test-slave-dmw7"
server_key: "13741b17c9232d3c5aa8ed6501bbfd7f"
server_key_iv: "db3310edfecda7cb571d07d1b0379071"
server_key_version: "encryptionkey@0"
I20260923 22:21:17.903052 22959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:17.906323 22959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:17.908398 22959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:17.908509 22959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:17.912307 22959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37287
I20260923 22:21:17.912375 23026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37287 every 8 connection(s)
I20260923 22:21:17.912715 22959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:17.913717 23027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:17.914840 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 22959
I20260923 22:21:17.914978 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:17.915150 1541 external_mini_cluster.cc:1428] Setting key 395e313de30907167082c74f2b91d755
I20260923 22:21:17.916819 23027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159: Bootstrap starting.
I20260923 22:21:17.917567 23027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:17.917853 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:17.915701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:36733 (local address 127.0.0.1:37287)
0923 22:21:17.915934 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:17.915944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:17.915974 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:21:17.916526 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:17.916535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:17.916571 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:17.916727 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:17.917375 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:17.917379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:17.917379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:17.917383 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:17.917421 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:17.917431 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:17.917505 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:17.917587 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 22:21:17.917645 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":60,"threads_started":1}
I20260923 22:21:17.917879 23027 log.cc:824] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:17.918748 23027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159: No bootstrap required, opened a new log
I20260923 22:21:17.918921 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:17.921578 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:17.920489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:50107 (local address 127.0.0.1:37287)
0923 22:21:17.920532 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:17.920534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:17.920618 (+ 84us) server_negotiation.cc:407] Connection header received
0923 22:21:17.920682 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:17.920682 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:17.920686 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:17.920732 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:17.921307 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:17.921308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:17.921308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:17.921311 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:17.921324 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:17.921328 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:17.921372 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:17.921424 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 22:21:17.921436 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:21:17.921479 23027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbe92692bf44e7fba2ec1f17460b159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } }
I20260923 22:21:17.921663 23027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:17.921700 23027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbe92692bf44e7fba2ec1f17460b159, State: Initialized, Role: FOLLOWER
I20260923 22:21:17.921834 23027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbe92692bf44e7fba2ec1f17460b159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } }
I20260923 22:21:17.921947 23027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:17.922010 23027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:17.922065 23027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:17.922622 23027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbe92692bf44e7fba2ec1f17460b159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } }
I20260923 22:21:17.922785 23027 leader_election.cc:302] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bbe92692bf44e7fba2ec1f17460b159; no voters:
I20260923 22:21:17.922945 23027 leader_election.cc:288] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:17.922972 23037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:17.923144 23037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [term 1 LEADER]: Becoming Leader. State: Replica: 5bbe92692bf44e7fba2ec1f17460b159, State: Running, Role: LEADER
I20260923 22:21:17.923193 23027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:17.923353 23037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbe92692bf44e7fba2ec1f17460b159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } }
I20260923 22:21:17.923722 23039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bbe92692bf44e7fba2ec1f17460b159. Latest consensus state: current_term: 1 leader_uuid: "5bbe92692bf44e7fba2ec1f17460b159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbe92692bf44e7fba2ec1f17460b159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } } }
I20260923 22:21:17.923861 23039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:17.923718 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bbe92692bf44e7fba2ec1f17460b159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbe92692bf44e7fba2ec1f17460b159" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37287 } } }
I20260923 22:21:17.924072 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:17.924110 23046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:17.924612 23046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:17.926329 23046 catalog_manager.cc:1347] Generated new cluster ID: 501cbb83d5874bdaa73a5395f6d0ffc5
I20260923 22:21:17.926398 23046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:17.933193 23046 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:17.933384 23046 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:17.940724 23046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bbe92692bf44e7fba2ec1f17460b159: Generated new TSK 0
I20260923 22:21:17.940958 23046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:17.942607 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 22959
2026-09-23T22:21:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:17 dist-test-slave-dmw7 krb5kdc[23058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:17 dist-test-slave-dmw7 krb5kdc[23058](info): set up 2 sockets
Sep 23 22:21:17 dist-test-slave-dmw7 krb5kdc[23058](info): commencing operation
krb5kdc: starting...
W20260923 22:21:18.978605 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:19 dist-test-slave-dmw7 krb5kdc[23058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:19.089180 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:40857
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:19.167811 23071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:19.168040 23071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:19.168087 23071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:19.169179 23071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:19.169238 23071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:19.169263 23071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:19.169292 23071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:19.169322 23071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:19.170812 23071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:40857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:19.171069 23071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:19.171281 23071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:19.176172 23079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:19.176327 23071 server_base.cc:1034] running on GCE node
W20260923 22:21:19.176162 23077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:19.176162 23076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:19.176649 23071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:19.176942 23071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:19.178081 23071 hybrid_clock.cc:648] HybridClock initialized: now 1790202079178061 us; error 37 us; skew 500 ppm
Sep 23 22:21:19 dist-test-slave-dmw7 krb5kdc[23058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:19.180400 23071 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:19.180836 23071 webserver.cc:460] Webserver started at http://127.1.129.126:36855/ using document root <none> and password file <none>
I20260923 22:21:19.181013 23071 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:19.181052 23071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:19.181136 23071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:19.182125 23071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "f1ad6e00822d4b63bcf1595ce55f4b26"
format_stamp: "Formatted at 2026-09-23 22:21:19 on dist-test-slave-dmw7"
server_key: "34a9da3c38b55c95a8ee216c0f5eb6b9"
server_key_iv: "1edaac85e4fedf7accef6050a0d7052a"
server_key_version: "encryptionkey@0"
I20260923 22:21:19.182423 23071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "f1ad6e00822d4b63bcf1595ce55f4b26"
format_stamp: "Formatted at 2026-09-23 22:21:19 on dist-test-slave-dmw7"
server_key: "34a9da3c38b55c95a8ee216c0f5eb6b9"
server_key_iv: "1edaac85e4fedf7accef6050a0d7052a"
server_key_version: "encryptionkey@0"
I20260923 22:21:19.183736 23071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:21:19.184403 23086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:19.184587 23071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:19.184674 23071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "f1ad6e00822d4b63bcf1595ce55f4b26"
format_stamp: "Formatted at 2026-09-23 22:21:19 on dist-test-slave-dmw7"
server_key: "34a9da3c38b55c95a8ee216c0f5eb6b9"
server_key_iv: "1edaac85e4fedf7accef6050a0d7052a"
server_key_version: "encryptionkey@0"
I20260923 22:21:19.184758 23071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:19.196640 23071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:19.198395 23071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:19.198544 23071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:19.202428 23071 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:40857
I20260923 22:21:19.202461 23138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:40857 every 8 connection(s)
I20260923 22:21:19.202896 23071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:19.203398 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23071
I20260923 22:21:19.203540 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:19.203712 1541 external_mini_cluster.cc:1428] Setting key 1e83f016129f76bf82c40b4625749c93
I20260923 22:21:19.204010 23139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:19.207386 23139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26: Bootstrap starting.
I20260923 22:21:19.208062 23139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:19.208353 23139 log.cc:824] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:19.209153 23139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26: No bootstrap required, opened a new log
Sep 23 22:21:19 dist-test-slave-dmw7 krb5kdc[23058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:19.211475 23139 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad6e00822d4b63bcf1595ce55f4b26" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40857 } }
I20260923 22:21:19.211648 23139 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:19.211687 23139 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ad6e00822d4b63bcf1595ce55f4b26, State: Initialized, Role: FOLLOWER
I20260923 22:21:19.211848 23139 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad6e00822d4b63bcf1595ce55f4b26" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40857 } }
I20260923 22:21:19.211930 23139 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:19.211951 23139 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:19.212009 23139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:19.212584 23139 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad6e00822d4b63bcf1595ce55f4b26" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40857 } }
I20260923 22:21:19.212656 23139 leader_election.cc:302] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1ad6e00822d4b63bcf1595ce55f4b26; no voters:
I20260923 22:21:19.212765 23139 leader_election.cc:288] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:19.212829 23144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:19.212962 23144 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [term 1 LEADER]: Becoming Leader. State: Replica: f1ad6e00822d4b63bcf1595ce55f4b26, State: Running, Role: LEADER
I20260923 22:21:19.212978 23139 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:19.213047 23144 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad6e00822d4b63bcf1595ce55f4b26" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40857 } }
I20260923 22:21:19.213377 23144 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1ad6e00822d4b63bcf1595ce55f4b26. Latest consensus state: current_term: 1 leader_uuid: "f1ad6e00822d4b63bcf1595ce55f4b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad6e00822d4b63bcf1595ce55f4b26" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40857 } } }
I20260923 22:21:19.213476 23144 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:19.213388 23145 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1ad6e00822d4b63bcf1595ce55f4b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1ad6e00822d4b63bcf1595ce55f4b26" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 40857 } } }
I20260923 22:21:19.213536 23145 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:19.213891 23152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:19.214357 23152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:19.214995 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:19.204305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50579 (local address 127.1.129.126:40857)
0923 22:21:19.204472 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:19.204480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:19.204579 (+ 99us) server_negotiation.cc:407] Connection header received
0923 22:21:19.205123 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:19.205129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:19.205189 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:19.205299 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:19.206078 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:19.206996 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:19.208024 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:19.208497 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:19.208575 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:19.211260 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:19.211264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:19.211264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:19.211274 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:19.212919 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:19.213420 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:19.213423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:19.213423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:19.213542 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:19.213828 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:19.213829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:19.213829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:19.214141 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:19.214336 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:19.214483 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 22:21:19.214796 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260923 22:21:19.216038 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:19.216246 23152 catalog_manager.cc:1347] Generated new cluster ID: 9d32a041b7574d028d0602df064880fd
I20260923 22:21:19.216296 23152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:19.222411 23152 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:19.223105 23152 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:19.224990 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:19.217430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40391 (local address 127.1.129.126:40857)
0923 22:21:19.217479 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:19.217481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:19.218885 (+ 1404us) server_negotiation.cc:407] Connection header received
0923 22:21:19.218960 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:19.218961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:19.218972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:19.219035 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:19.219546 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:19.220268 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:19.220977 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:19.221213 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:19.221294 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:19.222425 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:19.222426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:19.222426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:19.222433 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:19.223886 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:19.224235 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:19.224236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:19.224236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:19.224290 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:19.224503 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:19.224504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:19.224505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:19.224607 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:19.224708 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:19.224828 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 22:21:19.224887 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:21:19.234542 23152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1ad6e00822d4b63bcf1595ce55f4b26: Generated new TSK 0
I20260923 22:21:19.234742 23152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:19.241503 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23071
W20260923 22:21:19.245729 23056 connection.cc:535] client connection to 127.1.129.126:40857 recv error: Network error: failed to read from TLS socket (remote: 127.1.129.126:40857): Connection reset by peer (error 104)
2026-09-23T22:21:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:19 dist-test-slave-dmw7 krb5kdc[23170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:19 dist-test-slave-dmw7 krb5kdc[23170](info): set up 2 sockets
Sep 23 22:21:19 dist-test-slave-dmw7 krb5kdc[23170](info): commencing operation
krb5kdc: starting...
W20260923 22:21:20.274806 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:20 dist-test-slave-dmw7 krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:20.393505 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37335
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:37335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:20.473220 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:20.473407 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:20.473430 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:20.474566 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:20.474608 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:20.474622 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:20.474658 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:20.474694 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:20.476233 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:37335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:37335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:20.476464 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:20.476693 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:20.481523 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:20.481554 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:20.481575 23183 server_base.cc:1034] running on GCE node
W20260923 22:21:20.481546 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:20.481988 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:20.482276 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:20.483415 23183 hybrid_clock.cc:648] HybridClock initialized: now 1790202080483394 us; error 46 us; skew 500 ppm
Sep 23 22:21:20 dist-test-slave-dmw7 krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202080, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:20.491572 23183 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:20.492033 23183 webserver.cc:460] Webserver started at http://127.1.129.126:40863/ using document root <none> and password file <none>
I20260923 22:21:20.492293 23183 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:20.492362 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:20.492511 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:20.493595 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "f911ebd13348489b9952cfb3d507ab71"
format_stamp: "Formatted at 2026-09-23 22:21:20 on dist-test-slave-dmw7"
server_key: "620f480fc51847bf6e389b9f830719d8"
server_key_iv: "3d909950153a5927c4f34f193974685a"
server_key_version: "encryptionkey@0"
I20260923 22:21:20.493957 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "f911ebd13348489b9952cfb3d507ab71"
format_stamp: "Formatted at 2026-09-23 22:21:20 on dist-test-slave-dmw7"
server_key: "620f480fc51847bf6e389b9f830719d8"
server_key_iv: "3d909950153a5927c4f34f193974685a"
server_key_version: "encryptionkey@0"
I20260923 22:21:20.495426 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:20.496176 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:20.496346 23183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:20.496452 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "f911ebd13348489b9952cfb3d507ab71"
format_stamp: "Formatted at 2026-09-23 22:21:20 on dist-test-slave-dmw7"
server_key: "620f480fc51847bf6e389b9f830719d8"
server_key_iv: "3d909950153a5927c4f34f193974685a"
server_key_version: "encryptionkey@0"
I20260923 22:21:20.496526 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:20.505620 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:20.507921 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:20.508088 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:20.511973 23183 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:37335
I20260923 22:21:20.512004 23250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:37335 every 8 connection(s)
I20260923 22:21:20.512354 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:20.513230 23251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:20.515492 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71: Bootstrap starting.
I20260923 22:21:20.516001 23251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:20.516251 23251 log.cc:824] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:20.516903 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71: No bootstrap required, opened a new log
I20260923 22:21:20.518324 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23183
I20260923 22:21:20.518463 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:20.518616 1541 external_mini_cluster.cc:1428] Setting key 48256225ef326d954412b1b5a92d33f2
I20260923 22:21:20.518697 23251 raft_consensus.cc:348] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f911ebd13348489b9952cfb3d507ab71" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37335 } }
I20260923 22:21:20.518862 23251 raft_consensus.cc:374] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:20.518887 23251 raft_consensus.cc:729] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f911ebd13348489b9952cfb3d507ab71, State: Initialized, Role: FOLLOWER
I20260923 22:21:20.519090 23251 consensus_queue.cc:260] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f911ebd13348489b9952cfb3d507ab71" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37335 } }
I20260923 22:21:20.519197 23251 raft_consensus.cc:388] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:20.519243 23251 raft_consensus.cc:482] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:20.519338 23251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:20.520412 23251 raft_consensus.cc:504] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f911ebd13348489b9952cfb3d507ab71" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37335 } }
I20260923 22:21:20.520498 23251 leader_election.cc:302] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f911ebd13348489b9952cfb3d507ab71; no voters:
I20260923 22:21:20.520695 23251 leader_election.cc:288] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:20.520716 23256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:20.520818 23256 raft_consensus.cc:686] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [term 1 LEADER]: Becoming Leader. State: Replica: f911ebd13348489b9952cfb3d507ab71, State: Running, Role: LEADER
I20260923 22:21:20.520895 23256 consensus_queue.cc:237] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f911ebd13348489b9952cfb3d507ab71" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37335 } }
I20260923 22:21:20.520947 23251 sys_catalog.cc:564] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:20.521226 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f911ebd13348489b9952cfb3d507ab71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f911ebd13348489b9952cfb3d507ab71" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37335 } } }
I20260923 22:21:20.521297 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:20.521422 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f911ebd13348489b9952cfb3d507ab71. Latest consensus state: current_term: 1 leader_uuid: "f911ebd13348489b9952cfb3d507ab71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f911ebd13348489b9952cfb3d507ab71" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 37335 } } }
I20260923 22:21:20.521510 23261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:20.521575 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:20.522060 23261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:20.524044 23261 catalog_manager.cc:1347] Generated new cluster ID: 0e6e09a3d64e428bb467b0649501d0af
I20260923 22:21:20.524101 23261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:20.524513 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:20.519156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35519 (local address 127.1.129.126:37335)
0923 22:21:20.519348 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:20.519357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:20.519396 (+ 39us) server_negotiation.cc:407] Connection header received
0923 22:21:20.519547 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:20.519553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:20.519595 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:20.519708 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:20.519990 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:20.521113 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:20.522325 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:20.522946 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:20.523082 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:20.523812 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:20.523815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:20.523816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:20.524010 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:20.524048 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:20.524061 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:20.524155 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:20.524279 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 22:21:20.524309 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260923 22:21:20.525422 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:20.529531 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:20.526728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58525 (local address 127.1.129.126:37335)
0923 22:21:20.526764 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:20.526766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:20.526875 (+ 109us) server_negotiation.cc:407] Connection header received
0923 22:21:20.526933 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:20.526934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:20.526944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:20.526998 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:20.527145 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:20.527600 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:20.528343 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:20.528555 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:20.528636 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:20.529294 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:20.529295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:20.529295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:20.529300 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:20.529312 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:20.529316 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:20.529369 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:20.529429 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 22:21:20.529440 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:20.586488 23261 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:20.587289 23261 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:20.595237 23261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f911ebd13348489b9952cfb3d507ab71: Generated new TSK 0
I20260923 22:21:20.595444 23261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:20.599848 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23183
W20260923 22:21:20.604209 23168 connection.cc:535] client connection to 127.1.129.126:37335 recv error: Network error: failed to read from TLS socket (remote: 127.1.129.126:37335): Connection reset by peer (error 104)
2026-09-23T22:21:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:20 dist-test-slave-dmw7 krb5kdc[23282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:20 dist-test-slave-dmw7 krb5kdc[23282](info): set up 2 sockets
Sep 23 22:21:20 dist-test-slave-dmw7 krb5kdc[23282](info): commencing operation
krb5kdc: starting...
W20260923 22:21:21.626798 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:21 dist-test-slave-dmw7 krb5kdc[23282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:21.742985 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36135
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:21.821543 23295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:21.821786 23295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:21.821822 23295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:21.823022 23295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:21.823083 23295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:21.823105 23295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:21.823143 23295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:21.823174 23295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:21.824620 23295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:21.824894 23295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:21.825120 23295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:21.829901 23300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:21.829929 23301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:21.830068 23295 server_base.cc:1034] running on GCE node
W20260923 22:21:21.829908 23303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:21.830495 23295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:21.830819 23295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:21.831969 23295 hybrid_clock.cc:648] HybridClock initialized: now 1790202081831955 us; error 30 us; skew 500 ppm
Sep 23 22:21:21 dist-test-slave-dmw7 krb5kdc[23282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202081, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:21.834405 23295 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:21.834898 23295 webserver.cc:460] Webserver started at http://127.1.129.126:39555/ using document root <none> and password file <none>
I20260923 22:21:21.835136 23295 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:21.835207 23295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:21.835361 23295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:21.836391 23295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "4bbed5cc4d844ea38058c3c1902aed3d"
format_stamp: "Formatted at 2026-09-23 22:21:21 on dist-test-slave-dmw7"
server_key: "9bf70d5ff788b0786dfc086e157a172f"
server_key_iv: "cd668a239d4cb15507d6be5cecc2f870"
server_key_version: "encryptionkey@0"
I20260923 22:21:21.836748 23295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "4bbed5cc4d844ea38058c3c1902aed3d"
format_stamp: "Formatted at 2026-09-23 22:21:21 on dist-test-slave-dmw7"
server_key: "9bf70d5ff788b0786dfc086e157a172f"
server_key_iv: "cd668a239d4cb15507d6be5cecc2f870"
server_key_version: "encryptionkey@0"
I20260923 22:21:21.838044 23295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:21.838821 23310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:21.839010 23295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:21.839107 23295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "4bbed5cc4d844ea38058c3c1902aed3d"
format_stamp: "Formatted at 2026-09-23 22:21:21 on dist-test-slave-dmw7"
server_key: "9bf70d5ff788b0786dfc086e157a172f"
server_key_iv: "cd668a239d4cb15507d6be5cecc2f870"
server_key_version: "encryptionkey@0"
I20260923 22:21:21.839195 23295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:21.842311 23295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:21.844298 23295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:21.844456 23295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:21.848995 23295 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:36135
I20260923 22:21:21.849011 23362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:36135 every 8 connection(s)
I20260923 22:21:21.849438 23295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:21.851687 23363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:21.854475 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d: Bootstrap starting.
I20260923 22:21:21.855118 23363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:21.855402 23363 log.cc:824] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:21.856036 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d: No bootstrap required, opened a new log
I20260923 22:21:21.857100 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23295
I20260923 22:21:21.857208 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:21.857398 1541 external_mini_cluster.cc:1428] Setting key b1dd2775dda29a5247d622443f503d05
I20260923 22:21:21.857997 23363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbed5cc4d844ea38058c3c1902aed3d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36135 } }
I20260923 22:21:21.858186 23363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:21.858220 23363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbed5cc4d844ea38058c3c1902aed3d, State: Initialized, Role: FOLLOWER
I20260923 22:21:21.858356 23363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbed5cc4d844ea38058c3c1902aed3d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36135 } }
I20260923 22:21:21.858438 23363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:21.858470 23363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:21.858556 23363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:21.859589 23363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbed5cc4d844ea38058c3c1902aed3d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36135 } }
I20260923 22:21:21.859710 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:21.857935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52103 (local address 127.1.129.126:36135)
0923 22:21:21.858123 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:21.858133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:21.858164 (+ 31us) server_negotiation.cc:407] Connection header received
0923 22:21:21.858292 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:21.858299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:21.858336 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:21.858477 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:21.859249 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:21.859253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:21.859253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:21.859257 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:21.859302 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:21.859317 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:21.859394 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:21.859450 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 22:21:21.859488 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260923 22:21:21.859696 23363 leader_election.cc:302] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbed5cc4d844ea38058c3c1902aed3d; no voters:
I20260923 22:21:21.859853 23363 leader_election.cc:288] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:21.859874 23368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:21.859978 23368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [term 1 LEADER]: Becoming Leader. State: Replica: 4bbed5cc4d844ea38058c3c1902aed3d, State: Running, Role: LEADER
I20260923 22:21:21.860102 23363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:21.860109 23368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbed5cc4d844ea38058c3c1902aed3d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36135 } }
I20260923 22:21:21.860549 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:21.860514 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bbed5cc4d844ea38058c3c1902aed3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbed5cc4d844ea38058c3c1902aed3d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36135 } } }
I20260923 22:21:21.860592 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:21.860775 23374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:21.861312 23374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:21.862146 23370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bbed5cc4d844ea38058c3c1902aed3d. Latest consensus state: current_term: 1 leader_uuid: "4bbed5cc4d844ea38058c3c1902aed3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbed5cc4d844ea38058c3c1902aed3d" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36135 } } }
I20260923 22:21:21.862219 23370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:21.863739 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:21.862533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36743 (local address 127.1.129.126:36135)
0923 22:21:21.862572 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:21.862574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:21.862730 (+ 156us) server_negotiation.cc:407] Connection header received
0923 22:21:21.862826 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:21.862827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:21.862842 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:21.862907 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:21.863497 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:21.863498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:21.863498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:21.863502 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:21.863515 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:21.863520 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:21.863583 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:21.863670 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 22:21:21.863684 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:21.864181 23389 mvcc.cc:205] Tried to move back new op lower bound from 7332667727310442496 to 7332667727299715072. Current Snapshot: MvccSnapshot[applied={T|T < 7332667727310442496}]
I20260923 22:21:21.864391 23374 catalog_manager.cc:1347] Generated new cluster ID: 7d8c974698334168b7e6dd84d38b6a5b
I20260923 22:21:21.864432 23374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:21.874202 23374 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:21.874364 23374 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:21.877413 23374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bbed5cc4d844ea38058c3c1902aed3d: Generated new TSK 0
I20260923 22:21:21.877583 23374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:21.881332 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23295
2026-09-23T22:21:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:21 dist-test-slave-dmw7 krb5kdc[23394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:21 dist-test-slave-dmw7 krb5kdc[23394](info): set up 2 sockets
Sep 23 22:21:21 dist-test-slave-dmw7 krb5kdc[23394](info): commencing operation
krb5kdc: starting...
W20260923 22:21:22.906589 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:22 dist-test-slave-dmw7 krb5kdc[23394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:23.019763 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34971
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:34971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:23.098073 23407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:23.098263 23407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:23.098285 23407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:23.099342 23407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:23.099375 23407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:23.099388 23407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:23.099400 23407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:23.099412 23407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:23.100720 23407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:34971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:34971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:23.100888 23407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:23.101029 23407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:23.105474 23415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:23.105581 23412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:23.105633 23413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:23.105624 23407 server_base.cc:1034] running on GCE node
I20260923 22:21:23.105926 23407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:23.106220 23407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:23.107400 23407 hybrid_clock.cc:648] HybridClock initialized: now 1790202083107383 us; error 45 us; skew 500 ppm
Sep 23 22:21:23 dist-test-slave-dmw7 krb5kdc[23394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:23.109897 23407 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:23.110352 23407 webserver.cc:460] Webserver started at http://127.1.129.126:33405/ using document root <none> and password file <none>
I20260923 22:21:23.110591 23407 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:23.110673 23407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:23.110816 23407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:23.111840 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "9492b9610f344e62a29ff50820942685"
format_stamp: "Formatted at 2026-09-23 22:21:23 on dist-test-slave-dmw7"
server_key: "18cf3c75e67fdddf12308c42f745183d"
server_key_iv: "d92fd01d2b3020a0ebbf64ae1817c269"
server_key_version: "encryptionkey@0"
I20260923 22:21:23.112185 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "9492b9610f344e62a29ff50820942685"
format_stamp: "Formatted at 2026-09-23 22:21:23 on dist-test-slave-dmw7"
server_key: "18cf3c75e67fdddf12308c42f745183d"
server_key_iv: "d92fd01d2b3020a0ebbf64ae1817c269"
server_key_version: "encryptionkey@0"
I20260923 22:21:23.113417 23407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:23.114079 23422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:23.114244 23407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:23.114334 23407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "9492b9610f344e62a29ff50820942685"
format_stamp: "Formatted at 2026-09-23 22:21:23 on dist-test-slave-dmw7"
server_key: "18cf3c75e67fdddf12308c42f745183d"
server_key_iv: "d92fd01d2b3020a0ebbf64ae1817c269"
server_key_version: "encryptionkey@0"
I20260923 22:21:23.114408 23407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:23.133383 23407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:23.135731 23407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:23.135897 23407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:23.139640 23407 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:34971
I20260923 22:21:23.139686 23474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:34971 every 8 connection(s)
I20260923 22:21:23.140017 23407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:23.140908 23475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:23.143178 23475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685: Bootstrap starting.
I20260923 22:21:23.143724 23475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:23.144001 23475 log.cc:824] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:23.144256 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23407
I20260923 22:21:23.144394 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:23.144604 1541 external_mini_cluster.cc:1428] Setting key 32e5165fcc55f7f5381aa668dd6f3217
I20260923 22:21:23.144801 23475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685: No bootstrap required, opened a new log
I20260923 22:21:23.147015 23475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9492b9610f344e62a29ff50820942685" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34971 } }
I20260923 22:21:23.147223 23475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:23.147262 23475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9492b9610f344e62a29ff50820942685, State: Initialized, Role: FOLLOWER
I20260923 22:21:23.147413 23475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9492b9610f344e62a29ff50820942685" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34971 } }
I20260923 22:21:23.147501 23475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:23.147534 23475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:23.147578 23475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:23.148269 23475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9492b9610f344e62a29ff50820942685" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34971 } }
I20260923 22:21:23.148375 23475 leader_election.cc:302] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9492b9610f344e62a29ff50820942685; no voters:
I20260923 22:21:23.148530 23475 leader_election.cc:288] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:23.148598 23480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:23.148749 23475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:23.148774 23480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [term 1 LEADER]: Becoming Leader. State: Replica: 9492b9610f344e62a29ff50820942685, State: Running, Role: LEADER
I20260923 22:21:23.148859 23480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9492b9610f344e62a29ff50820942685" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34971 } }
I20260923 22:21:23.149261 23480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9492b9610f344e62a29ff50820942685. Latest consensus state: current_term: 1 leader_uuid: "9492b9610f344e62a29ff50820942685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9492b9610f344e62a29ff50820942685" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34971 } } }
I20260923 22:21:23.149478 23480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:23.149410 23481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9492b9610f344e62a29ff50820942685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9492b9610f344e62a29ff50820942685" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 34971 } } }
I20260923 22:21:23.149876 23481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685 [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:23.150554 23496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:23.150619 23496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:23.150689 23489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:23.150870 23489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:23.152433 23489 catalog_manager.cc:1347] Generated new cluster ID: 0c75e2b375b14492832b9ebe40f13bd8
I20260923 22:21:23.152480 23489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:21:23 dist-test-slave-dmw7 krb5kdc[23394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:23.157226 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:23.145146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55345 (local address 127.1.129.126:34971)
0923 22:21:23.145332 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:23.145342 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:23.145376 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:23.146171 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:23.146180 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:23.146243 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:23.146398 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:23.147108 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:23.148788 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:23.149998 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:23.150660 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:23.151126 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:23.153623 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:23.153627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:23.153627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:23.153638 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:23.155610 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:23.156017 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:23.156020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:23.156020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:23.156120 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:23.156336 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:23.156336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:23.156336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:23.156616 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:23.156756 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:23.156911 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 22:21:23.157034 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260923 22:21:23.158217 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:23.164639 23489 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:23.165241 23489 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:23.166726 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:23.159722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60145 (local address 127.1.129.126:34971)
0923 22:21:23.159778 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:23.159779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:23.159976 (+ 197us) server_negotiation.cc:407] Connection header received
0923 22:21:23.160032 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:23.160033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:23.160043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:23.160109 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:23.160663 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:23.161378 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:23.162350 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:23.162692 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:23.162754 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:23.164035 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:23.164037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:23.164037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:23.164044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:23.165606 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:23.165962 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:23.165963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:23.165964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:23.166023 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:23.166211 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:23.166212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:23.166212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:23.166331 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:23.166423 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:23.166525 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 22:21:23.166617 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:21:23.169410 23489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9492b9610f344e62a29ff50820942685: Generated new TSK 0
I20260923 22:21:23.169597 23489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:23.174574 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23407
2026-09-23T22:21:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:23 dist-test-slave-dmw7 krb5kdc[23506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:23 dist-test-slave-dmw7 krb5kdc[23506](info): set up 2 sockets
Sep 23 22:21:23 dist-test-slave-dmw7 krb5kdc[23506](info): commencing operation
krb5kdc: starting...
W20260923 22:21:24.222822 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:24 dist-test-slave-dmw7 krb5kdc[23506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:24.338979 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:33959
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:33959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:24.418885 23519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:24.419067 23519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:24.419092 23519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:24.420224 23519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:24.420274 23519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:24.420289 23519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:24.420303 23519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:24.420315 23519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:24.421761 23519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:33959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:33959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:24.421994 23519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:24.422209 23519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:24.426937 23525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:24.427000 23527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:24.427134 23524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:24.427134 23519 server_base.cc:1034] running on GCE node
I20260923 22:21:24.427413 23519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:24.427707 23519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:24.428874 23519 hybrid_clock.cc:648] HybridClock initialized: now 1790202084428870 us; error 36 us; skew 500 ppm
Sep 23 22:21:24 dist-test-slave-dmw7 krb5kdc[23506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:24.431291 23519 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:24.431694 23519 webserver.cc:460] Webserver started at http://127.1.129.126:46457/ using document root <none> and password file <none>
I20260923 22:21:24.431927 23519 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:24.432001 23519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:24.432148 23519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:24.433130 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "1bced047ae734c69bf4607a964742ffa"
format_stamp: "Formatted at 2026-09-23 22:21:24 on dist-test-slave-dmw7"
server_key: "175928db039d98aecb2c95799961adfc"
server_key_iv: "f594bbdd6220f7af69ef6389d9f149a4"
server_key_version: "encryptionkey@0"
I20260923 22:21:24.433475 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "1bced047ae734c69bf4607a964742ffa"
format_stamp: "Formatted at 2026-09-23 22:21:24 on dist-test-slave-dmw7"
server_key: "175928db039d98aecb2c95799961adfc"
server_key_iv: "f594bbdd6220f7af69ef6389d9f149a4"
server_key_version: "encryptionkey@0"
I20260923 22:21:24.434785 23519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:21:24.435461 23534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:24.435632 23519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:24.435727 23519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "1bced047ae734c69bf4607a964742ffa"
format_stamp: "Formatted at 2026-09-23 22:21:24 on dist-test-slave-dmw7"
server_key: "175928db039d98aecb2c95799961adfc"
server_key_iv: "f594bbdd6220f7af69ef6389d9f149a4"
server_key_version: "encryptionkey@0"
I20260923 22:21:24.435811 23519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:24.454363 23519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:24.456225 23519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:24.456388 23519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:24.460273 23519 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:33959
I20260923 22:21:24.460319 23586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:33959 every 8 connection(s)
I20260923 22:21:24.460696 23519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:24.461562 23587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:24.463517 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23519
I20260923 22:21:24.463614 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:24.463800 1541 external_mini_cluster.cc:1428] Setting key 3d7302f129b7b284e106bf53b34b87d6
I20260923 22:21:24.463845 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa: Bootstrap starting.
I20260923 22:21:24.464501 23587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:24.464803 23587 log.cc:824] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:24.465463 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa: No bootstrap required, opened a new log
I20260923 22:21:24.467240 23587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bced047ae734c69bf4607a964742ffa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33959 } }
I20260923 22:21:24.467391 23587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:24.467455 23587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bced047ae734c69bf4607a964742ffa, State: Initialized, Role: FOLLOWER
I20260923 22:21:24.467615 23587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bced047ae734c69bf4607a964742ffa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33959 } }
I20260923 22:21:24.467710 23587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:24.467753 23587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:24.467809 23587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:24.468456 23587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bced047ae734c69bf4607a964742ffa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33959 } }
I20260923 22:21:24.468580 23587 leader_election.cc:302] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bced047ae734c69bf4607a964742ffa; no voters:
I20260923 22:21:24.468730 23587 leader_election.cc:288] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:24.468750 23592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:24.468859 23592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [term 1 LEADER]: Becoming Leader. State: Replica: 1bced047ae734c69bf4607a964742ffa, State: Running, Role: LEADER
I20260923 22:21:24.468936 23592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bced047ae734c69bf4607a964742ffa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33959 } }
I20260923 22:21:24.468981 23587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:24.469242 23592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bced047ae734c69bf4607a964742ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bced047ae734c69bf4607a964742ffa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33959 } } }
I20260923 22:21:24.469377 23592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:24.469542 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:24.464338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35601 (local address 127.1.129.126:33959)
0923 22:21:24.464517 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:24.464524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:24.464547 (+ 23us) server_negotiation.cc:407] Connection header received
0923 22:21:24.465052 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:24.465059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:24.465106 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:24.465248 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:24.465468 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:24.466377 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:24.467247 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:24.467763 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:24.467866 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:24.468658 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:24.468663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:24.468663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:24.468891 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:24.468936 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:24.468951 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:24.469020 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:24.469213 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 22:21:24.469285 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260923 22:21:24.469595 23592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bced047ae734c69bf4607a964742ffa. Latest consensus state: current_term: 1 leader_uuid: "1bced047ae734c69bf4607a964742ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bced047ae734c69bf4607a964742ffa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 33959 } } }
I20260923 22:21:24.470052 23592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:24.470449 23599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:24.470646 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:24.470947 23599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:24.472649 23599 catalog_manager.cc:1347] Generated new cluster ID: 38eca3fbd4ff42aeb1911233fcd3a4bb
I20260923 22:21:24.472700 23599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:24.475101 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:24.472162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40987 (local address 127.1.129.126:33959)
0923 22:21:24.472204 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:24.472207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:24.472362 (+ 155us) server_negotiation.cc:407] Connection header received
0923 22:21:24.472421 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:24.472422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:24.472431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:24.472481 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:24.472655 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:24.473146 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:24.473827 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:24.474056 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:24.474148 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:24.474811 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:24.474813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:24.474813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:24.474819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:24.474833 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:24.474837 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:24.474896 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:24.474960 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 22:21:24.474981 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:24.495142 23599 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:24.495754 23599 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:24.504107 23599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bced047ae734c69bf4607a964742ffa: Generated new TSK 0
I20260923 22:21:24.504308 23599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:24.506368 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23519
2026-09-23T22:21:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:24 dist-test-slave-dmw7 krb5kdc[23618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:24 dist-test-slave-dmw7 krb5kdc[23618](info): set up 2 sockets
Sep 23 22:21:24 dist-test-slave-dmw7 krb5kdc[23618](info): commencing operation
krb5kdc: starting...
W20260923 22:21:25.526949 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:25 dist-test-slave-dmw7 krb5kdc[23618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:25.641302 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36065
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:36065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:25.720125 23631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:25.720336 23631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:25.720360 23631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:25.721516 23631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:25.721560 23631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:25.721575 23631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:25.721587 23631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:25.721599 23631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:25.723158 23631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:36065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:36065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:25.723397 23631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:25.723608 23631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:25.728633 23639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:25.728638 23636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:25.728667 23631 server_base.cc:1034] running on GCE node
W20260923 22:21:25.728642 23637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:25.729080 23631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:25.729377 23631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:25.730551 23631 hybrid_clock.cc:648] HybridClock initialized: now 1790202085730538 us; error 42 us; skew 500 ppm
Sep 23 22:21:25 dist-test-slave-dmw7 krb5kdc[23618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:25.737618 23631 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:25.738112 23631 webserver.cc:460] Webserver started at http://127.1.129.126:41187/ using document root <none> and password file <none>
I20260923 22:21:25.738348 23631 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:25.738420 23631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:25.738576 23631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:25.739647 23631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "1fd108cea21c46e89505465fd9c5a052"
format_stamp: "Formatted at 2026-09-23 22:21:25 on dist-test-slave-dmw7"
server_key: "b4ccc4155fbb5b3afdb9957989308b36"
server_key_iv: "9985bb5a704cfa8508a77a32fca461d4"
server_key_version: "encryptionkey@0"
I20260923 22:21:25.740010 23631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "1fd108cea21c46e89505465fd9c5a052"
format_stamp: "Formatted at 2026-09-23 22:21:25 on dist-test-slave-dmw7"
server_key: "b4ccc4155fbb5b3afdb9957989308b36"
server_key_iv: "9985bb5a704cfa8508a77a32fca461d4"
server_key_version: "encryptionkey@0"
I20260923 22:21:25.741390 23631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:25.742156 23646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:25.742347 23631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:25.742449 23631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "1fd108cea21c46e89505465fd9c5a052"
format_stamp: "Formatted at 2026-09-23 22:21:25 on dist-test-slave-dmw7"
server_key: "b4ccc4155fbb5b3afdb9957989308b36"
server_key_iv: "9985bb5a704cfa8508a77a32fca461d4"
server_key_version: "encryptionkey@0"
I20260923 22:21:25.742532 23631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:25.745786 23631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:25.747822 23631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:25.747978 23631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:25.752022 23631 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:36065
I20260923 22:21:25.752050 23698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:36065 every 8 connection(s)
I20260923 22:21:25.752377 23631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:25.753342 23699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:25.755430 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23631
I20260923 22:21:25.755548 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:25.755738 1541 external_mini_cluster.cc:1428] Setting key 9ee6ee3f75917110d793bf53a31aa11c
I20260923 22:21:25.755864 23699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052: Bootstrap starting.
I20260923 22:21:25.756562 23699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:25.756870 23699 log.cc:824] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:25.757737 23699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052: No bootstrap required, opened a new log
I20260923 22:21:25.758718 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:25.756307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37729 (local address 127.1.129.126:36065)
0923 22:21:25.756501 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:25.756511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:25.756550 (+ 39us) server_negotiation.cc:407] Connection header received
0923 22:21:25.757166 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:25.757174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:25.757209 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:25.757388 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:25.758160 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:25.758164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:25.758164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:25.758168 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:25.758213 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:25.758228 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:25.758323 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:25.758453 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 22:21:25.758494 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260923 22:21:25.759701 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:25.760152 23699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd108cea21c46e89505465fd9c5a052" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36065 } }
I20260923 22:21:25.760334 23699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:25.760366 23699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd108cea21c46e89505465fd9c5a052, State: Initialized, Role: FOLLOWER
I20260923 22:21:25.760495 23699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd108cea21c46e89505465fd9c5a052" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36065 } }
I20260923 22:21:25.760584 23699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:25.760614 23699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:25.760661 23699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:25.761426 23699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd108cea21c46e89505465fd9c5a052" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36065 } }
I20260923 22:21:25.761550 23699 leader_election.cc:302] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fd108cea21c46e89505465fd9c5a052; no voters:
I20260923 22:21:25.761751 23699 leader_election.cc:288] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:25.761961 23710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:25.761987 23699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:25.762262 23710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [term 1 LEADER]: Becoming Leader. State: Replica: 1fd108cea21c46e89505465fd9c5a052, State: Running, Role: LEADER
I20260923 22:21:25.762395 23710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd108cea21c46e89505465fd9c5a052" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36065 } }
I20260923 22:21:25.762482 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:25.761244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33371 (local address 127.1.129.126:36065)
0923 22:21:25.761298 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:25.761300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:25.761366 (+ 66us) server_negotiation.cc:407] Connection header received
0923 22:21:25.761440 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:25.761441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:25.761445 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:25.761514 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:25.762153 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:25.762154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:25.762154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:25.762158 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:25.762170 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:25.762190 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:25.762395 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:25.762414 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 22:21:25.762426 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:21:25.762984 23710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fd108cea21c46e89505465fd9c5a052. Latest consensus state: current_term: 1 leader_uuid: "1fd108cea21c46e89505465fd9c5a052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd108cea21c46e89505465fd9c5a052" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36065 } } }
I20260923 22:21:25.763064 23710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:25.763324 23709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fd108cea21c46e89505465fd9c5a052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd108cea21c46e89505465fd9c5a052" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 36065 } } }
I20260923 22:21:25.763477 23709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052 [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:25.764058 23724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:25.764097 23724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:25.764168 23722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:25.764305 23722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:25.765606 23722 catalog_manager.cc:1347] Generated new cluster ID: 5d80a05e52cf44d9b2cd8a145aae7176
I20260923 22:21:25.765656 23722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:25.779030 23722 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:25.779241 23722 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:25.782413 23722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fd108cea21c46e89505465fd9c5a052: Generated new TSK 0
I20260923 22:21:25.782622 23722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:25.793534 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23631
2026-09-23T22:21:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1287 ms)
[----------] 18 tests from AuthTokenIssuingTest (23469 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:25 dist-test-slave-dmw7 krb5kdc[23730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:25 dist-test-slave-dmw7 krb5kdc[23730](info): set up 2 sockets
Sep 23 22:21:25 dist-test-slave-dmw7 krb5kdc[23730](info): commencing operation
krb5kdc: starting...
W20260923 22:21:26.823107 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:26 dist-test-slave-dmw7 krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:26.939253 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44123
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:44123,127.1.129.125:41823,127.1.129.124:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:27.018935 23743 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:27.019197 23743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:27.019227 23743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:27.019271 23743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:27.020390 23743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:27.020460 23743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:27.020484 23743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:27.020512 23743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:27.020543 23743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:27.021996 23743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:44123,127.1.129.125:41823,127.1.129.124:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:27.022294 23743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:27.022518 23743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:27.027295 23748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:27.027343 23749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:27.027384 23751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:27.027662 23743 server_base.cc:1034] running on GCE node
I20260923 22:21:27.027843 23743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:27.028061 23743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:27.029251 23743 hybrid_clock.cc:648] HybridClock initialized: now 1790202087029234 us; error 31 us; skew 500 ppm
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:27.031876 23743 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:27.032366 23743 webserver.cc:460] Webserver started at http://127.1.129.126:43957/ using document root <none> and password file <none>
I20260923 22:21:27.032627 23743 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:27.032698 23743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:27.032866 23743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:27.033938 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "c05f78428d8448b5be28e91307574bfa"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "cfb260534d55173da3c8e023f47af5dc"
server_key_iv: "fb87d8a62bfa7425a479e94d1fe92bca"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.034406 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "c05f78428d8448b5be28e91307574bfa"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "cfb260534d55173da3c8e023f47af5dc"
server_key_iv: "fb87d8a62bfa7425a479e94d1fe92bca"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.036010 23743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:27.036773 23758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:27.036957 23743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:27.037050 23743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "c05f78428d8448b5be28e91307574bfa"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "cfb260534d55173da3c8e023f47af5dc"
server_key_iv: "fb87d8a62bfa7425a479e94d1fe92bca"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.037142 23743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:27.051671 23743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:27.054194 23743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:27.054374 23743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:27.058730 23743 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:44123
I20260923 22:21:27.058809 23810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:44123 every 8 connection(s)
I20260923 22:21:27.059150 23743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:27.060170 23811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:27.061713 23811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:27.063591 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23743
I20260923 22:21:27.063704 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:27.063908 1541 external_mini_cluster.cc:1428] Setting key e5984a79677f3d1789e2ca09de50dff6
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.129.125@KRBTEST.COM; defaulting to no policy
I20260923 22:21:27.072922 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.062366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44123 (local address 127.0.0.1:41896)
0923 22:21:27.062534 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.062570 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.063066 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.063325 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.063327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.063560 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.063751 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.063756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.065319 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.065321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.066440 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.066442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.067052 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.067053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.067135 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.067640 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.067650 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.069315 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.071330 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.071336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.071338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.071558 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.071814 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.071815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.071815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.071906 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.072291 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.072296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.072355 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.072496 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.072742 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":100,"thread_start_us":63,"threads_started":1}
I20260923 22:21:27.072926 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.062511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.1.129.126:44123)
0923 22:21:27.062653 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.062658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.062692 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:27.063201 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.063202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.063218 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.063293 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.063851 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.065218 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.066534 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.066944 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.067038 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.069425 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.069433 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.069433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.069446 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.071200 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.071650 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.071654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.071654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.071740 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.071989 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.071990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.071990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.072216 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.072314 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.072531 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.072677 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.125@KRBTEST.COM" created.
I20260923 22:21:27.073810 23811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:27.074206 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.074036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38250)
0923 22:21:27.074116 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.074169 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 22:21:27.074373 23759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
I20260923 22:21:27.074868 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.074755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38252)
0923 22:21:27.074803 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.074836 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:21:27.075054 23811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:41823: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab.
Entry for principal kudu/127.1.129.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab.
I20260923 22:21:27.087848 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.125:41823
--webserver_interface=127.1.129.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab
--principal=kudu/127.1.129.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:44123,127.1.129.125:41823,127.1.129.124:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:21:27.124322 23811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } attempt: 1
I20260923 22:21:27.124850 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.124708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38260)
0923 22:21:27.124777 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.124819 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 22:21:27.125240 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.125129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38268)
0923 22:21:27.125181 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.125213 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 22:21:27.125344 23811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:41823: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
W20260923 22:21:27.166831 23818 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:27.167065 23818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:27.167095 23818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:27.167161 23818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:27.168309 23818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:27.168375 23818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:27.168401 23818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:27.168430 23818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:27.168462 23818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:27.169915 23818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:44123,127.1.129.125:41823,127.1.129.124:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab
--principal=kudu/127.1.129.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.125:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.129.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:27.170194 23818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:27.170408 23818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:27.175292 23826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:27.175283 23824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:27.175323 23823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:27.175661 23818 server_base.cc:1034] running on GCE node
I20260923 22:21:27.175880 23818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:27.176105 23818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:27.177278 23818 hybrid_clock.cc:648] HybridClock initialized: now 1790202087177268 us; error 39 us; skew 500 ppm
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:27.179777 23818 init.cc:377] Logged in from keytab as kudu/127.1.129.125@KRBTEST.COM (short username kudu)
I20260923 22:21:27.180215 23818 webserver.cc:460] Webserver started at http://127.1.129.125:38515/ using document root <none> and password file <none>
I20260923 22:21:27.180435 23818 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:27.180503 23818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:27.180661 23818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:27.181692 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/data/instance:
uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "7eccd0a5501879b4310c060e6a328d78"
server_key_iv: "580b1f2cb63cddba8fed8fb523acb46e"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.182068 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/wal/instance:
uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "7eccd0a5501879b4310c060e6a328d78"
server_key_iv: "580b1f2cb63cddba8fed8fb523acb46e"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.183450 23818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:27.184172 23833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:27.184331 23818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:27.184435 23818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/wal
uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "7eccd0a5501879b4310c060e6a328d78"
server_key_iv: "580b1f2cb63cddba8fed8fb523acb46e"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.184511 23818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:27.193485 23811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } attempt: 2
I20260923 22:21:27.193913 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.193762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38284)
0923 22:21:27.193824 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.193872 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 22:21:27.194286 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.194196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38300)
0923 22:21:27.194242 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.194269 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 22:21:27.194396 23811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:41823: Network error: Client connection negotiation failed: client connection to 127.1.129.125:41823: connect: Connection refused (error 111)
I20260923 22:21:27.202847 23818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:27.205361 23818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:27.205540 23818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:27.210031 23818 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.125:41823
I20260923 22:21:27.210064 23885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.125:41823 every 8 connection(s)
I20260923 22:21:27.210465 23818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
I20260923 22:21:27.211529 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:27.212507 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23818
I20260923 22:21:27.212612 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-1/wal/instance
I20260923 22:21:27.212777 1541 external_mini_cluster.cc:1428] Setting key 54e6fa8f7a32539e1b262c244018a752
I20260923 22:21:27.213444 23886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.129.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.124@KRBTEST.COM" created.
I20260923 22:21:27.223917 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.214270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.1.129.126:44123)
0923 22:21:27.214317 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.214320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.214604 (+ 284us) server_negotiation.cc:407] Connection header received
0923 22:21:27.215330 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.215331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.215345 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.215400 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.216228 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.216699 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.218315 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.218512 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.218573 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.220939 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.220941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.220942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.220950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.222436 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.222907 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.222908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.222908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.222962 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.223180 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.223181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.223181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.223274 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.223321 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.223687 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.223804 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:21:27.224054 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.214267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44123 (local address 127.0.0.1:41904)
0923 22:21:27.214447 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.214498 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.215201 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.215477 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.215481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.215859 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.216145 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.216151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.216790 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.216791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.218227 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.218230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.218609 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.218611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.218793 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.219299 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.219329 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.220827 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.222523 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.222529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.222531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.222839 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.223027 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.223028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.223028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.223137 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.223350 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.223356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.223470 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.223642 (+ 172us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.223797 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260923 22:21:27.224617 23886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:21:27.232797 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.224880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38312)
0923 22:21:27.224953 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.224965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.225010 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.225307 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.225308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.225470 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.225530 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.225532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.226725 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.226726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.227325 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.227331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.228021 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.228022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.228071 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.228510 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.228513 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.229611 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.231800 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.231801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.231801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.231967 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.232166 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.232167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.232167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.232219 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.232494 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.232495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.232545 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.232621 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.232715 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 22:21:27.232899 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.224951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.1.129.125:41823)
0923 22:21:27.225173 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.225177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.225193 (+ 16us) server_negotiation.cc:407] Connection header received
0923 22:21:27.225211 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.225212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.225223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.225282 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.225642 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.226596 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.227419 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.227943 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.228028 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.229693 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.229719 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.229719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.229726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.231715 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.232045 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.232048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.232049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.232100 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.232260 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.232260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.232260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.232379 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.232503 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.232643 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.232762 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
I20260923 22:21:27.233508 23886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:27.233930 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.233809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50642)
0923 22:21:27.233861 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.233904 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 22:21:27.234055 23836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
I20260923 22:21:27.234488 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.234381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50652)
0923 22:21:27.234443 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.234467 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 22:21:27.234620 23886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:41385: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab.
Entry for principal kudu/127.1.129.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab.
I20260923 22:21:27.239977 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.124:41385
--webserver_interface=127.1.129.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab
--principal=kudu/127.1.129.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:44123,127.1.129.125:41823,127.1.129.124:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:21:27.283860 23886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } attempt: 1
I20260923 22:21:27.284437 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.284290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50656)
0923 22:21:27.284348 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.284402 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 22:21:27.284829 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.284750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50660)
0923 22:21:27.284791 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.284815 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:21:27.284933 23886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:41385: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
I20260923 22:21:27.285522 23811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } attempt: 3
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:21:27.292660 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.285786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38326)
0923 22:21:27.285831 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.285841 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.285885 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.286023 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.286024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.286219 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.286287 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.286289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.287106 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.287107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.287693 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.287695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.288083 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.288084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.288174 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.288638 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.288643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.289988 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.291604 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.291605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.291605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.291772 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.291983 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.291984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.291984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.292040 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.292328 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.292329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.292375 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.292469 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.292566 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 22:21:27.292660 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.285823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.1.129.125:41823)
0923 22:21:27.285860 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.285862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.285872 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:27.285949 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.285950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.285961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.286016 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.286372 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.287010 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.287768 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.287997 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.288083 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.290073 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.290075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.290075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.290082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.291504 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.291856 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.291858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.291858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.291916 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.292127 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.292128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.292129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.292242 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.292334 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.292493 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.292566 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:27.293383 23811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:27.293829 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.293721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50664)
0923 22:21:27.293772 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.293800 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 22:21:27.294162 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.294088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50680)
0923 22:21:27.294111 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.294148 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260923 22:21:27.294266 23811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:41385: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
W20260923 22:21:27.322031 23893 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:27.322242 23893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:27.322258 23893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:27.322278 23893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:27.323385 23893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:27.323423 23893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:27.323438 23893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:27.323450 23893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:27.323462 23893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:27.324877 23893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:44123,127.1.129.125:41823,127.1.129.124:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab
--principal=kudu/127.1.129.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.124:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.129.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:27.325057 23893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:27.325217 23893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:21:27.325456 23811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } attempt: 1
I20260923 22:21:27.325989 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.325827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50692)
0923 22:21:27.325912 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.325958 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 22:21:27.326538 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.326433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50698)
0923 22:21:27.326482 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.326506 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:21:27.326678 23811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:41385: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
W20260923 22:21:27.330355 23898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:27.330348 23899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:27.330365 23901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:27.330528 23893 server_base.cc:1034] running on GCE node
I20260923 22:21:27.330900 23893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:27.331223 23893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:27.332393 23893 hybrid_clock.cc:648] HybridClock initialized: now 1790202087332368 us; error 43 us; skew 500 ppm
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:27.334937 23893 init.cc:377] Logged in from keytab as kudu/127.1.129.124@KRBTEST.COM (short username kudu)
I20260923 22:21:27.335390 23893 webserver.cc:460] Webserver started at http://127.1.129.124:34731/ using document root <none> and password file <none>
I20260923 22:21:27.335616 23893 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:27.335667 23893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:27.335784 23893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:27.336846 23893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/data/instance:
uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "d7af28aa0b86d46713d169f1742df175"
server_key_iv: "7e584dfdcd129ca49d112d1258873389"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.337209 23893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/wal/instance:
uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "d7af28aa0b86d46713d169f1742df175"
server_key_iv: "7e584dfdcd129ca49d112d1258873389"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.338760 23893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:27.339828 23908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:27.339988 23893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:27.340091 23893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/wal
uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb"
format_stamp: "Formatted at 2026-09-23 22:21:27 on dist-test-slave-dmw7"
server_key: "d7af28aa0b86d46713d169f1742df175"
server_key_iv: "7e584dfdcd129ca49d112d1258873389"
server_key_version: "encryptionkey@0"
I20260923 22:21:27.340178 23893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:27.353103 23886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } attempt: 2
I20260923 22:21:27.353193 23893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:27.353612 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.353468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50708)
0923 22:21:27.353536 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.353583 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 22:21:27.354007 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.353891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50714)
0923 22:21:27.353951 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.353983 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 22:21:27.354125 23886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:41385: Network error: Client connection negotiation failed: client connection to 127.1.129.124:41385: connect: Connection refused (error 111)
I20260923 22:21:27.355911 23893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:27.356029 23893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:27.359838 23893 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.124:41385
I20260923 22:21:27.359915 23960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.124:41385 every 8 connection(s)
I20260923 22:21:27.360262 23893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
I20260923 22:21:27.361344 23961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:27.362977 23961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:27.364920 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 23893
I20260923 22:21:27.365000 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201819212015-1541-0/minicluster-data/master-2/wal/instance
I20260923 22:21:27.365195 1541 external_mini_cluster.cc:1428] Setting key fd85028021acfe4d39fb43db5e07db5f
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:21:27.373610 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.363750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.1.129.126:44123)
0923 22:21:27.363820 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.363821 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.364154 (+ 333us) server_negotiation.cc:407] Connection header received
0923 22:21:27.364658 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.364659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.364671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.364728 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.365583 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.366234 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.367379 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.367656 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.367711 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.370183 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.370185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.370186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.370202 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.371962 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.372474 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.372475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.372476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.372530 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.372823 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.372823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.372823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.372932 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.372982 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.373399 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.373497 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:21:27.373685 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.363736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:44123 (local address 127.0.0.1:41920)
0923 22:21:27.363920 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.363956 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.364549 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.364818 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.364821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.365172 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.365482 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.365488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.366327 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.366328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.367292 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.367294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.367752 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.367753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.367888 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.368414 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.368424 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.370064 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.372042 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.372049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.372052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.372390 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.372606 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.372607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.372607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.372747 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.373000 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.373006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.373175 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.373311 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.373479 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":20,"thread_start_us":87,"threads_started":1}
I20260923 22:21:27.374253 23961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:27.375113 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.365598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.1.129.124:41385)
0923 22:21:27.365788 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.365792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.365811 (+ 19us) server_negotiation.cc:407] Connection header received
0923 22:21:27.365832 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.365834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.365852 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.365936 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.366598 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.367492 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.368443 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.368744 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.368800 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.371360 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.371369 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.371369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.371384 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.373309 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.373772 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.373775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.373776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.373841 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.374158 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.374159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.374159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.374321 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.374481 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.374908 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.375029 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:27.381852 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.374610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.1.129.125:41823)
0923 22:21:27.374688 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.374690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.374702 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:21:27.374733 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.374734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.374746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.374813 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.375171 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.375770 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.376538 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.376769 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.376848 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.379005 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.379007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.379007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.379014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.380465 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.380848 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.380849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.380850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.380902 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.381095 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.381096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.381097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.381209 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.381262 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.381727 (+ 465us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.381790 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 22:21:27.382133 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.374544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:41823 (local address 127.0.0.1:38332)
0923 22:21:27.374586 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.374595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.374681 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.374838 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.374840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.375036 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.375086 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.375087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.375872 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.375873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.376447 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.376449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.376846 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.376847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.376919 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.377376 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.377381 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.378896 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.380545 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.380546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.380547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.380769 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.380976 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.380977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.380977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.381037 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.381555 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.381556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.381608 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.381695 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.381785 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 22:21:27.382666 23961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:27.384107 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.376013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.1.129.126:44123)
0923 22:21:27.376056 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.376058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.376074 (+ 16us) server_negotiation.cc:407] Connection header received
0923 22:21:27.376098 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.376099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.376109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.376170 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.376677 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.377238 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.378158 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.378474 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.378561 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.381052 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.381053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.381054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.381061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.382872 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.383344 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.383345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.383345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.383406 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.383616 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.383617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.383617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.383727 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.383906 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.383942 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.384010 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:21:27.389830 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.382991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.1.129.124:41385)
0923 22:21:27.383050 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.383051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.383059 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:27.383080 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.383081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.383090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.383153 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.383368 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.383838 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.384717 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.385035 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.385105 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.387050 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.387051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.387052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.387058 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.388742 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.389094 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.389095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.389095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.389141 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.389332 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.389333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.389333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.389426 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.389484 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.389627 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.389734 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":37}
I20260923 22:21:27.389957 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.382940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50736)
0923 22:21:27.382985 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.382996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.383031 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.383155 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.383155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.383267 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.383297 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.383298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.383914 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.383916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.384634 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.384636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.385109 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.385110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.385155 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.385540 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.385545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.386969 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.388831 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.388831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.388832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.388987 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.389208 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.389209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.389209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.389271 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.389489 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.389490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.389533 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.389611 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.389816 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":109}
I20260923 22:21:27.391878 23961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb: Bootstrap starting.
I20260923 22:21:27.392925 23961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:27.393002 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.384684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.1.129.125:41823)
0923 22:21:27.384729 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.384731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.384739 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:27.384783 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.384784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.384790 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.384829 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.385436 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.386138 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.387119 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.387423 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.387490 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.390163 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.390164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.390165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.390171 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.391860 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.392252 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.392253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.392253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.392323 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.392508 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.392509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.392509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.392636 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.392791 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.392831 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.392907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 22:21:27.393260 23961 log.cc:824] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:27.394088 23961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb: No bootstrap required, opened a new log
I20260923 22:21:27.394475 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:27.395699 23961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } }
I20260923 22:21:27.395859 23961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b1d8e72d6a4f4d82b81d0ed39d86bb, State: Initialized, Role: FOLLOWER
I20260923 22:21:27.395947 23961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } }
I20260923 22:21:27.396271 23961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:27.396225 23967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.395543 23785 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c05f78428d8448b5be28e91307574bfa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41934
I20260923 22:21:27.396378 23967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:21:27.397957 23978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:27.398010 23978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:27.398310 23893 master_runner.cc:186] Error getting master registration for 127.1.129.126:44123: OK, instance_id { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" instance_seqno: 1790202087057494 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:21:27.401173 23893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:21:27.401834 23811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } attempt: 2
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:21:27.408491 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.402174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.1.129.124:41385)
0923 22:21:27.402216 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.402218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.402227 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:21:27.402288 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.402289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.402298 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.402353 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.402684 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.403144 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.403852 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.404071 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.404145 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.405922 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.405924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.405924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.405931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.407348 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.407695 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.407697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.407697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.407753 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.407951 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.407952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.407953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.408087 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.408160 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.408324 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.408400 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:27.408491 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.402130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50748)
0923 22:21:27.402171 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.402182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.402227 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.402358 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.402359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.402519 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.402558 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.402559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.403245 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.403246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.403758 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.403759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.404154 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.404155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.404214 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.404678 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.404684 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.405836 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.407447 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.407448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.407449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.407621 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.407820 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.407821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.407821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.407879 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.408174 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.408175 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.408219 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.408301 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.408399 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260923 22:21:27.409713 23811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa: Bootstrap starting.
I20260923 22:21:27.410339 23811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:27.410712 23811 log.cc:824] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:27.411515 23811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa: No bootstrap required, opened a new log
I20260923 22:21:27.412496 23811 raft_consensus.cc:348] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } }
I20260923 22:21:27.412600 23811 raft_consensus.cc:729] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f78428d8448b5be28e91307574bfa, State: Initialized, Role: FOLLOWER
I20260923 22:21:27.412708 23811 consensus_queue.cc:260] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } }
I20260923 22:21:27.412951 23981 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.413018 23811 sys_catalog.cc:564] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:27.413024 23981 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:27.413142 23785 raft_consensus.cc:482] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 22:21:27.413230 23785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:27.413978 23785 raft_consensus.cc:504] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } }
I20260923 22:21:27.414229 23785 leader_election.cc:288] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [CANDIDATE]: Term 1 election: Requested vote from peers f2c8bb8ec2ae491ab5d59c96240e2b57 (127.1.129.125:41823), 99b1d8e72d6a4f4d82b81d0ed39d86bb (127.1.129.124:41385)
I20260923 22:21:27.414495 23935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c05f78428d8448b5be28e91307574bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb"
I20260923 22:21:27.414618 23935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:27.415685 23935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f78428d8448b5be28e91307574bfa in term 1.
I20260923 22:21:27.416585 23761 leader_election.cc:302] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99b1d8e72d6a4f4d82b81d0ed39d86bb, c05f78428d8448b5be28e91307574bfa; no voters:
I20260923 22:21:27.416731 23981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:27.416864 23981 raft_consensus.cc:686] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [term 1 LEADER]: Becoming Leader. State: Replica: c05f78428d8448b5be28e91307574bfa, State: Running, Role: LEADER
I20260923 22:21:27.416937 23981 consensus_queue.cc:237] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } }
I20260923 22:21:27.417316 23981 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c05f78428d8448b5be28e91307574bfa. Latest consensus state: current_term: 1 leader_uuid: "c05f78428d8448b5be28e91307574bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.417387 23981 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:27.416013 23996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:27.417670 23996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:27.417840 24003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:27.418041 24003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:27.417122 23743 master_runner.cc:186] Error getting master registration for 127.1.129.125:41823: OK, instance_id { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" instance_seqno: 1790202087208750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:21:27.419592 23743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:21:27.421099 23935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [term 1 FOLLOWER]: Refusing update from remote peer c05f78428d8448b5be28e91307574bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:27.421494 24001 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:27.423190 23967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c05f78428d8448b5be28e91307574bfa. Latest consensus state: current_term: 1 leader_uuid: "c05f78428d8448b5be28e91307574bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.423303 23967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:27.424881 23981 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c05f78428d8448b5be28e91307574bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.424958 23981 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:27.425402 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.415499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.1.129.125:41823)
0923 22:21:27.415552 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.415554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.416678 (+ 1124us) server_negotiation.cc:407] Connection header received
0923 22:21:27.416840 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.416841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.416853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.416886 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.418126 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.418873 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.419957 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.420730 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.420809 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.422413 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.422415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.422415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.422423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.424113 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.424520 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.424521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.424522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.424576 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.424777 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.424779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.424779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.424909 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.424997 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.425219 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.425303 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:21:27.425405 23967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c05f78428d8448b5be28e91307574bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.425479 23967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:27.426054 24004 mvcc.cc:205] Tried to move back new op lower bound from 7332667750073540608 to 7332667750060347392. Current Snapshot: MvccSnapshot[applied={T|T < 7332667750073540608}]
I20260923 22:21:27.426985 24005 mvcc.cc:205] Tried to move back new op lower bound from 7332667750073540608 to 7332667750060347392. Current Snapshot: MvccSnapshot[applied={T|T < 7332667750073540608}]
I20260923 22:21:27.427870 24003 catalog_manager.cc:1347] Generated new cluster ID: 936b1dee21de43ba936937967bc10587
I20260923 22:21:27.427915 24003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:27.433595 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.416333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.1.129.126:44123)
0923 22:21:27.416709 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.416711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.426163 (+ 9452us) server_negotiation.cc:407] Connection header received
0923 22:21:27.426256 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.426257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.426268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.426314 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.427018 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.427751 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.428760 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.429056 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.429096 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.430590 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.430592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.430592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.430599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.432334 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.432719 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.432720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.432720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.432775 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.432977 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.432978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.432979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.433114 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.433217 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.433418 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.433493 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260923 22:21:27.438446 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.415934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.1.129.124:41385)
0923 22:21:27.415979 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.415981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.426386 (+ 10405us) server_negotiation.cc:407] Connection header received
0923 22:21:27.426720 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.426722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.426733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.426767 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.429038 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.429772 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.433193 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.433500 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.433541 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.435562 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.435564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.435564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.435572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.437197 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.437590 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.437591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.437591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.437646 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.437844 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.437845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.437845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.437972 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.438053 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.438274 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.438352 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:21:27.445230 23886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } attempt: 3
Sep 23 22:21:27 dist-test-slave-dmw7 krb5kdc[23730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:21:27.453305 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.445517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:41385 (local address 127.0.0.1:50766)
0923 22:21:27.445569 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:27.445581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:27.445622 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:27.445760 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:27.445760 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:27.445931 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:27.445977 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.445979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.446897 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.446899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.447690 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:27.447691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.448167 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.448168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:27.448224 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.448643 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:27.448648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:27.450143 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:27.452228 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.452229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.452229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.452436 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.452645 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:27.452646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:27.452646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:27.452702 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:27.453005 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:27.453006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:27.453061 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 22:21:27.453107 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 22:21:27.453198 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:21:27.454694 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:27.445544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.1.129.124:41385)
0923 22:21:27.445593 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.445595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.445605 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:27.445673 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.445674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.445696 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.445754 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.446048 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.446819 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.447794 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:27.448097 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:27.448181 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:27.450256 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:27.450257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:27.450258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:27.450264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:27.452132 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.452532 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.452533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.452533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.452584 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:27.452784 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:27.452785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:27.452785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:27.452920 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:27.453484 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:27.454431 (+ 947us) server_negotiation.cc:299] Negotiation successful
0923 22:21:27.454551 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:21:27.456396 23886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57: Bootstrap starting.
I20260923 22:21:27.457142 23886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:27.457469 23886 log.cc:824] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:27.467319 23886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57: No bootstrap required, opened a new log
I20260923 22:21:27.469347 23886 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } }
I20260923 22:21:27.469494 23886 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c8bb8ec2ae491ab5d59c96240e2b57, State: Initialized, Role: FOLLOWER
I20260923 22:21:27.469588 23886 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } }
I20260923 22:21:27.469962 23886 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:27.470033 23860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c05f78428d8448b5be28e91307574bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57"
I20260923 22:21:27.470096 23859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:27.471093 23859 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [term 1 FOLLOWER]: Refusing update from remote peer c05f78428d8448b5be28e91307574bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:27.471889 23860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05f78428d8448b5be28e91307574bfa in term 1.
I20260923 22:21:27.472651 24008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:27.474908 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c05f78428d8448b5be28e91307574bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.474987 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:27.475033 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c05f78428d8448b5be28e91307574bfa. Latest consensus state: current_term: 1 leader_uuid: "c05f78428d8448b5be28e91307574bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.475095 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:27.475131 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c05f78428d8448b5be28e91307574bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.475188 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:27.477633 24003 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:27.478359 24003 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:27.483148 24024 mvcc.cc:205] Tried to move back new op lower bound from 7332667750073540608 to 7332667750060347392. Current Snapshot: MvccSnapshot[applied={T|T < 7332667750073540608}]
W20260923 22:21:27.483769 24022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:27.483886 24022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:27.483996 24001 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c05f78428d8448b5be28e91307574bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05f78428d8448b5be28e91307574bfa" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 44123 } } peers { permanent_uuid: "f2c8bb8ec2ae491ab5d59c96240e2b57" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 41823 } } peers { permanent_uuid: "99b1d8e72d6a4f4d82b81d0ed39d86bb" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 41385 } } }
I20260923 22:21:27.484107 24001 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:27.490605 24003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c05f78428d8448b5be28e91307574bfa: Generated new TSK 0
I20260923 22:21:27.490876 24003 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:21:27.525434 23814 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.524052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.1.129.126:44123)
0923 22:21:27.524097 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.524099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.524760 (+ 661us) server_negotiation.cc:407] Connection header received
0923 22:21:27.524868 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.524870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.524881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.524910 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.525363 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41948: BlockingRecv error: recv got EOF from 127.0.0.1:41948 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 22:21:27.526412 23965 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.524472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.1.129.124:41385)
0923 22:21:27.524510 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.524512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.525866 (+ 1354us) server_negotiation.cc:407] Connection header received
0923 22:21:27.525966 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.525967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.525977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.526007 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.526349 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50782: BlockingRecv error: recv got EOF from 127.0.0.1:50782 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 22:21:27.527609 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.526931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.1.129.125:41823)
0923 22:21:27.526979 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.526982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.527074 (+ 92us) server_negotiation.cc:407] Connection header received
0923 22:21:27.527181 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.527182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.527194 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.527225 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.527547 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38360: BlockingRecv error: recv got EOF from 127.0.0.1:38360 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260923 22:21:27.530049 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.529271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.1.129.125:41823)
0923 22:21:27.529306 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.529308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.529543 (+ 235us) server_negotiation.cc:407] Connection header received
0923 22:21:27.529646 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.529647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.529665 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.529697 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.530006 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38366: BlockingRecv error: recv got EOF from 127.0.0.1:38366 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:21:27.533149 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.532440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.1.129.125:41823)
0923 22:21:27.532473 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.532475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.532681 (+ 206us) server_negotiation.cc:407] Connection header received
0923 22:21:27.532777 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.532778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.532788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.532816 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.533110 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38374: BlockingRecv error: recv got EOF from 127.0.0.1:38374 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 22:21:27.537122 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.536386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.1.129.125:41823)
0923 22:21:27.536419 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.536421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.536630 (+ 209us) server_negotiation.cc:407] Connection header received
0923 22:21:27.536725 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.536726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.536736 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.536764 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.537081 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38386: BlockingRecv error: recv got EOF from 127.0.0.1:38386 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 22:21:27.543275 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.542392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.1.129.125:41823)
0923 22:21:27.542428 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.542429 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.542754 (+ 325us) server_negotiation.cc:407] Connection header received
0923 22:21:27.542856 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.542857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.542867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.542896 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.543233 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38402: BlockingRecv error: recv got EOF from 127.0.0.1:38402 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:21:27.553308 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.552544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.1.129.125:41823)
0923 22:21:27.552577 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.552580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.552784 (+ 204us) server_negotiation.cc:407] Connection header received
0923 22:21:27.552875 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.552876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.552885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.552915 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.553248 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38410: BlockingRecv error: recv got EOF from 127.0.0.1:38410 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 22:21:27.571853 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.570901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.1.129.125:41823)
0923 22:21:27.570947 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.570950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.571227 (+ 277us) server_negotiation.cc:407] Connection header received
0923 22:21:27.571342 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.571343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.571355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.571389 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.571794 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38414: BlockingRecv error: recv got EOF from 127.0.0.1:38414 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 22:21:27.606067 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.605303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38418 (local address 127.1.129.125:41823)
0923 22:21:27.605349 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.605352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.605493 (+ 141us) server_negotiation.cc:407] Connection header received
0923 22:21:27.605563 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.605564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.605576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.605644 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.605993 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38418: BlockingRecv error: recv got EOF from 127.0.0.1:38418 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 22:21:27.672392 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.671544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.1.129.125:41823)
0923 22:21:27.671622 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.671624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.671742 (+ 118us) server_negotiation.cc:407] Connection header received
0923 22:21:27.671842 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.671843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.671854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.671917 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.672321 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38432: BlockingRecv error: recv got EOF from 127.0.0.1:38432 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 22:21:27.802381 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:27.801676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.1.129.125:41823)
0923 22:21:27.801782 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:27.801784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:27.801797 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:21:27.801867 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:27.801868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:27.801880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:27.801940 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:27.802288 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38440: BlockingRecv error: recv got EOF from 127.0.0.1:38440 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260923 22:21:28.060411 23891 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:28.059713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.1.129.125:41823)
0923 22:21:28.059788 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:28.059791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:28.059821 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:21:28.059901 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:28.059902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:28.059913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:28.059989 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:28.060357 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38444: BlockingRecv error: recv got EOF from 127.0.0.1:38444 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 22:21:28.398458 23978 catalog_manager.cc:1259] Loaded cluster ID: 936b1dee21de43ba936937967bc10587
I20260923 22:21:28.398535 23978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb: loading cluster ID for follower catalog manager: success
I20260923 22:21:28.399362 23978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb: acquiring CA information for follower catalog manager: success
I20260923 22:21:28.399605 23978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99b1d8e72d6a4f4d82b81d0ed39d86bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:21:28.484387 24022 catalog_manager.cc:1259] Loaded cluster ID: 936b1dee21de43ba936937967bc10587
I20260923 22:21:28.484467 24022 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57: loading cluster ID for follower catalog manager: success
I20260923 22:21:28.485270 24022 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57: acquiring CA information for follower catalog manager: success
I20260923 22:21:28.485514 24022 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2c8bb8ec2ae491ab5d59c96240e2b57: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:21:28.576129 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:28.573519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.1.129.125:41823)
0923 22:21:28.573634 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:28.573636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:28.573646 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:28.573697 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:28.573698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:28.573744 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:28.573827 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:28.574029 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.574526 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.575362 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.575558 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.575619 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:28.575793 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:28.575908 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:28.575976 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:28.576015 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 22:21:28.576035 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260923 22:21:28.580632 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:28.577987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.1.129.124:41385)
0923 22:21:28.578108 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:28.578110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:28.578120 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:28.578194 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:28.578195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:28.578238 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:28.578326 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:28.578457 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.579060 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.579804 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.580023 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.580101 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:28.580282 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:28.580396 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:28.580484 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:28.580509 (+ 25us) server_negotiation.cc:299] Negotiation successful
0923 22:21:28.580531 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260923 22:21:28.585085 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:28.582484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.1.129.125:41823)
0923 22:21:28.582534 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:28.582536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:28.582714 (+ 178us) server_negotiation.cc:407] Connection header received
0923 22:21:28.582773 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:28.582774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:28.582787 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:28.582835 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:28.582968 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.583677 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.584404 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.584621 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.584693 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:28.584842 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:28.584874 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:28.584921 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:28.584977 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 22:21:28.584997 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:21:28.585517 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:28.582429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.1.129.124:41385)
0923 22:21:28.582467 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:28.582469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:28.582613 (+ 144us) server_negotiation.cc:407] Connection header received
0923 22:21:28.582682 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:28.582682 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:28.582695 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:28.582755 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:28.582905 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.583636 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.584725 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.585019 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.585058 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:28.585269 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:28.585298 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:28.585326 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:28.585418 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 22:21:28.585445 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:28.585683 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:28.582344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.1.129.126:44123)
0923 22:21:28.582716 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:28.582718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:28.582726 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:28.582772 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:28.582773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:28.582820 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:28.582906 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:28.583084 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.583561 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.584722 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.585026 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.585240 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:28.585310 (+ 70us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:28.585423 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:28.585513 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:28.585561 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 22:21:28.585583 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":58,"threads_started":1}
I20260923 22:21:28.587445 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23743
I20260923 22:21:28.593055 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23818
I20260923 22:21:28.598243 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 23893
2026-09-23T22:21:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2806 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:28 dist-test-slave-dmw7 krb5kdc[24111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:28 dist-test-slave-dmw7 krb5kdc[24111](info): set up 2 sockets
Sep 23 22:21:28 dist-test-slave-dmw7 krb5kdc[24111](info): commencing operation
krb5kdc: starting...
W20260923 22:21:29.626822 1541 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/test-user.keytab.
Sep 23 22:21:29 dist-test-slave-dmw7 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.129.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
Entry for principal kudu/127.1.129.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab.
I20260923 22:21:29.741566 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:39479
--webserver_interface=127.1.129.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:39479,127.1.129.125:40045,127.1.129.124:35475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:29.822191 24124 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:29.822439 24124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:29.822471 24124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:29.822520 24124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:29.823745 24124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:29.823819 24124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:29.823846 24124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:29.823879 24124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:29.823910 24124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:29.825402 24124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:39479,127.1.129.125:40045,127.1.129.124:35475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.126.keytab
--principal=kudu/127.1.129.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.126:39479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.129.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:29.825688 24124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:29.825917 24124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:29.831182 24129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:29.831194 24132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:29.831440 24124 server_base.cc:1034] running on GCE node
W20260923 22:21:29.831214 24130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:29.832661 24124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:29.832960 24124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:29.834110 24124 hybrid_clock.cc:648] HybridClock initialized: now 1790202089834087 us; error 37 us; skew 500 ppm
Sep 23 22:21:29 dist-test-slave-dmw7 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:29.836603 24124 init.cc:377] Logged in from keytab as kudu/127.1.129.126@KRBTEST.COM (short username kudu)
I20260923 22:21:29.836989 24124 webserver.cc:460] Webserver started at http://127.1.129.126:41007/ using document root <none> and password file <none>
I20260923 22:21:29.837170 24124 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:29.837211 24124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:29.837296 24124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:29.838264 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/data/instance:
uuid: "ffb334943d25412899a4ba25648352b8"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-dmw7"
server_key: "693f610f1ec6f9e57e41adfb37c74302"
server_key_iv: "712d2b914d33a8c4a0ede0c662f9e6c1"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.838621 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/wal/instance:
uuid: "ffb334943d25412899a4ba25648352b8"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-dmw7"
server_key: "693f610f1ec6f9e57e41adfb37c74302"
server_key_iv: "712d2b914d33a8c4a0ede0c662f9e6c1"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.839962 24124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:29.840685 24139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:29.840876 24124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:29.840979 24124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/wal
uuid: "ffb334943d25412899a4ba25648352b8"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-dmw7"
server_key: "693f610f1ec6f9e57e41adfb37c74302"
server_key_iv: "712d2b914d33a8c4a0ede0c662f9e6c1"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.841051 24124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:29.864091 24124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:29.866094 24124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:29.866253 24124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:29.870354 24124 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.126:39479
I20260923 22:21:29.870396 24191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.126:39479 every 8 connection(s)
I20260923 22:21:29.870855 24124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:29.871872 24192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:29.873374 24192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:29.876137 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 24124
I20260923 22:21:29.876219 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-0/wal/instance
I20260923 22:21:29.876406 1541 external_mini_cluster.cc:1428] Setting key 43154b2534ecd3cf546b87d11ded6928
Sep 23 22:21:29 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.129.125@KRBTEST.COM; defaulting to no policy
I20260923 22:21:29.884122 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:29.874082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.1.129.126:39479)
0923 22:21:29.874273 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:29.874277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:29.874305 (+ 28us) server_negotiation.cc:407] Connection header received
0923 22:21:29.874916 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:29.874918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:29.874937 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:29.875025 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:29.875539 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:29.876394 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:29.877729 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:29.878240 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:29.878366 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:29.880729 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:29.880737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:29.880737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:29.880744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:29.882377 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:29.882862 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:29.882866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:29.882866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:29.882953 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:29.883205 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:29.883206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:29.883206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:29.883412 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:29.883510 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:29.883716 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 22:21:29.883931 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260923 22:21:29.884130 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:29.874030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39479 (local address 127.0.0.1:35512)
0923 22:21:29.874195 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:29.874237 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:29.874764 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:29.875043 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:29.875045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:29.875302 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:29.875453 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:29.875458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:29.876497 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:29.876498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:29.877632 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:29.877636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:29.878350 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:29.878351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:29.878404 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:29.878976 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:29.878985 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:29.880622 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:29.882484 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:29.882490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:29.882492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:29.882769 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:29.883027 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:29.883028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:29.883029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:29.883129 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:29.883489 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:29.883494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:29.883555 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 22:21:29.883667 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 22:21:29.883879 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":26,"thread_start_us":68,"threads_started":1}
I20260923 22:21:29.884950 24192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:29.885381 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:29.885247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38460)
0923 22:21:29.885314 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:29.885358 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 22:21:29.885577 24142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
I20260923 22:21:29.886147 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:29.886028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38462)
0923 22:21:29.886072 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:29.886106 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:21:29.886304 24192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:40045: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab.
Entry for principal kudu/127.1.129.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab.
I20260923 22:21:29.899971 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.125:40045
--webserver_interface=127.1.129.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab
--principal=kudu/127.1.129.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:39479,127.1.129.125:40045,127.1.129.124:35475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:21:29.935570 24192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } attempt: 1
I20260923 22:21:29.936167 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:29.935977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38464)
0923 22:21:29.936052 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:29.936111 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 22:21:29.936635 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:29.936514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38468)
0923 22:21:29.936567 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:29.936604 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 22:21:29.936744 24192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:40045: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
W20260923 22:21:29.981788 24199 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:29.982005 24199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:29.982028 24199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:29.982051 24199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:29.983140 24199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:29.983179 24199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:29.983196 24199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:29.983210 24199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:29.983225 24199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:29.984647 24199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:39479,127.1.129.125:40045,127.1.129.124:35475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.125.keytab
--principal=kudu/127.1.129.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.125:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.129.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:29.984833 24199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:29.985077 24199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:29.989791 24207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:29.989815 24205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:29.989802 24204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:29.990099 24199 server_base.cc:1034] running on GCE node
I20260923 22:21:29.990368 24199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:29.990681 24199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:29.991835 24199 hybrid_clock.cc:648] HybridClock initialized: now 1790202089991821 us; error 42 us; skew 500 ppm
Sep 23 22:21:29 dist-test-slave-dmw7 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:29.994359 24199 init.cc:377] Logged in from keytab as kudu/127.1.129.125@KRBTEST.COM (short username kudu)
I20260923 22:21:29.994894 24199 webserver.cc:460] Webserver started at http://127.1.129.125:40269/ using document root <none> and password file <none>
I20260923 22:21:29.995162 24199 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:29.995221 24199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:29.995354 24199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:29.996379 24199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/data/instance:
uuid: "524c2a74940d48f58613f53a50e41270"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-dmw7"
server_key: "32339ca2479257c0b5d963b29177b01c"
server_key_iv: "e0a1377b1357af7ec2af1219305d4ba2"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.996742 24199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/wal/instance:
uuid: "524c2a74940d48f58613f53a50e41270"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-dmw7"
server_key: "32339ca2479257c0b5d963b29177b01c"
server_key_iv: "e0a1377b1357af7ec2af1219305d4ba2"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.998040 24199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:21:29.998906 24214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:29.999102 24199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:29.999191 24199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/wal
uuid: "524c2a74940d48f58613f53a50e41270"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-dmw7"
server_key: "32339ca2479257c0b5d963b29177b01c"
server_key_iv: "e0a1377b1357af7ec2af1219305d4ba2"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.999249 24199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:30.004889 24192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } attempt: 2
I20260923 22:21:30.005355 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.005224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38484)
0923 22:21:30.005282 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.005323 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:21:30.005752 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.005668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38490)
0923 22:21:30.005708 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.005734 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 22:21:30.005885 24192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.125:40045: Network error: Client connection negotiation failed: client connection to 127.1.129.125:40045: connect: Connection refused (error 111)
I20260923 22:21:30.007961 24199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:30.010363 24199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:30.010530 24199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:30.014744 24199 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.125:40045
I20260923 22:21:30.014789 24266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.125:40045 every 8 connection(s)
I20260923 22:21:30.015156 24199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/data/info.pb
I20260923 22:21:30.015416 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 24199
I20260923 22:21:30.015514 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-1/wal/instance
I20260923 22:21:30.015764 1541 external_mini_cluster.cc:1428] Setting key 1819b6886db87dea9ff34998bb5d9a36
I20260923 22:21:30.016352 24267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:30.018270 24267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.129.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.129.124@KRBTEST.COM" created.
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:30.029770 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.018990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.1.129.126:39479)
0923 22:21:30.019056 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.019058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.019233 (+ 175us) server_negotiation.cc:407] Connection header received
0923 22:21:30.019677 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.019678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.019693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.019769 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.020360 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.021203 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.022149 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.022492 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.022573 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.026766 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.026768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.026768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.026778 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.028288 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.028702 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.028704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.028704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.028777 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.029049 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.029050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.029051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.029225 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.029313 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.029589 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.029679 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:21:30.029955 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.018952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39479 (local address 127.0.0.1:35518)
0923 22:21:30.019101 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.019140 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.019583 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.019833 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.019836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.020066 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.020268 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.020271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.021299 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.021300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.022057 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.022058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.022593 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.022594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.022805 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.023357 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.023367 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.026614 (+ 3247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.028377 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.028383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.028385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.028598 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.028866 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.028867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.028867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.028967 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.029320 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.029326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.029439 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.029567 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.029728 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260923 22:21:30.030498 24267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:21:30.038885 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.030875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.1.129.125:40045)
0923 22:21:30.031047 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.031051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.031066 (+ 15us) server_negotiation.cc:407] Connection header received
0923 22:21:30.031082 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.031083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.031093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.031171 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.031440 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.032087 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.032758 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.033046 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.033134 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.035978 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.035985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.035985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.035991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.037596 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.037986 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.037988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.037988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.038037 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.038278 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.038279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.038279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.038446 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.038519 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.038682 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.038785 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260923 22:21:30.038954 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.030788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38498)
0923 22:21:30.030860 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.030869 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.030910 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.031190 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.031190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.031321 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.031362 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.031363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.032176 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.032176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.032668 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.032669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.033132 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.033133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.033205 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.033595 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.033600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.035878 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.037711 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.037712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.037712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.037915 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.038095 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.038096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.038096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.038146 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.038514 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.038514 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.038552 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.038607 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.038868 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":77}
I20260923 22:21:30.039633 24267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:30.039983 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.039872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36054)
0923 22:21:30.039922 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.039964 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 22:21:30.040115 24216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
I20260923 22:21:30.040525 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.040440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36058)
0923 22:21:30.040487 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.040510 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 22:21:30.040642 24267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:35475: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.129.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab.
Entry for principal kudu/127.1.129.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab.
I20260923 22:21:30.041384 1541 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDHvgV0/build/release/bin/kudu
/tmp/dist-test-taskDHvgV0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.124:35475
--webserver_interface=127.1.129.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab
--principal=kudu/127.1.129.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.129.84:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.129.126:39479,127.1.129.125:40045,127.1.129.124:35475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:21:30.089859 24267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } attempt: 1
I20260923 22:21:30.090384 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.090251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36064)
0923 22:21:30.090313 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.090355 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 22:21:30.090787 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.090702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36076)
0923 22:21:30.090744 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.090769 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:21:30.090905 24267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:35475: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
I20260923 22:21:30.097072 24192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } attempt: 3
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:21:30.104099 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.097475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.1.129.125:40045)
0923 22:21:30.097533 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.097535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.097546 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:21:30.097606 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.097607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.097630 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.097696 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.098014 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.098497 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.099261 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.099465 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.099536 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.101424 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.101425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.101426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.101432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.102972 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.103315 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.103317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.103317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.103372 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.103579 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.103580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.103581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.103702 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.103772 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.103938 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.104010 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 22:21:30.104101 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.097404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38504)
0923 22:21:30.097457 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.097467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.097539 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.097702 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.097703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.097868 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.097935 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.097936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.098581 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.098582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.099170 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.099172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.099533 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.099534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.099611 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.100075 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.100080 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.101326 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.103047 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.103048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.103049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.103231 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.103436 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.103437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.103437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.103514 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.103778 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.103779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.103824 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.103902 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.104003 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:21:30.104432 24192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:30.104766 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.104672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36086)
0923 22:21:30.104720 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.104744 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:21:30.105185 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.105080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36088)
0923 22:21:30.105148 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.105170 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 22:21:30.105288 24192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:35475: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
W20260923 22:21:30.121002 24274 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:30.121200 24274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:30.121217 24274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:30.121237 24274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:30.122259 24274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:30.122291 24274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:30.122305 24274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:30.122341 24274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:30.122354 24274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:30.123796 24274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.129.84:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.129.126:39479,127.1.129.125:40045,127.1.129.124:35475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/krb5kdc/kudu_127.1.129.124.keytab
--principal=kudu/127.1.129.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.129.124:35475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.129.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 20:53:37 UTC on ccycloud.quasar-rkviyx.root.comops.site
build id 4668083
I20260923 22:21:30.123974 24274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:30.124122 24274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:30.128772 24282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:30.128732 24280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:30.128916 24274 server_base.cc:1034] running on GCE node
W20260923 22:21:30.128772 24279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:30.129331 24274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:30.129632 24274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:30.130771 24274 hybrid_clock.cc:648] HybridClock initialized: now 1790202090130755 us; error 38 us; skew 500 ppm
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:30.133263 24274 init.cc:377] Logged in from keytab as kudu/127.1.129.124@KRBTEST.COM (short username kudu)
I20260923 22:21:30.133741 24274 webserver.cc:460] Webserver started at http://127.1.129.124:37599/ using document root <none> and password file <none>
I20260923 22:21:30.133955 24274 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:30.133999 24274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:30.134153 24274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:30.135212 24274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/data/instance:
uuid: "d274b4138dc445eb9f289003f744bb9d"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-dmw7"
server_key: "2b6c99fe856a2f8996cb12c27e332d57"
server_key_iv: "08c4c3eb0b6cf58a4e2ba9ca72a160a5"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.135560 24274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/wal/instance:
uuid: "d274b4138dc445eb9f289003f744bb9d"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-dmw7"
server_key: "2b6c99fe856a2f8996cb12c27e332d57"
server_key_iv: "08c4c3eb0b6cf58a4e2ba9ca72a160a5"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.136415 24192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } attempt: 1
I20260923 22:21:30.136816 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.136677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36092)
0923 22:21:30.136751 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.136788 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 22:21:30.136958 24274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:30.137236 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.137101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36094)
0923 22:21:30.137149 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.137204 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:21:30.137344 24192 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:35475: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
I20260923 22:21:30.137799 24289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:30.138001 24274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:30.138088 24274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/data,/tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/wal
uuid: "d274b4138dc445eb9f289003f744bb9d"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-dmw7"
server_key: "2b6c99fe856a2f8996cb12c27e332d57"
server_key_iv: "08c4c3eb0b6cf58a4e2ba9ca72a160a5"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.138154 24274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:30.157024 24274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:30.159070 24267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } attempt: 2
I20260923 22:21:30.159305 24274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:30.159482 24274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:30.159637 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.159485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36100)
0923 22:21:30.159557 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.159599 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 22:21:30.160040 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.159905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36116)
0923 22:21:30.159970 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.160009 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 22:21:30.160194 24267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.129.124:35475: Network error: Client connection negotiation failed: client connection to 127.1.129.124:35475: connect: Connection refused (error 111)
I20260923 22:21:30.163518 24274 rpc_server.cc:307] RPC server started. Bound to: 127.1.129.124:35475
I20260923 22:21:30.163568 24341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.129.124:35475 every 8 connection(s)
I20260923 22:21:30.163878 24274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/data/info.pb
I20260923 22:21:30.164780 24342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:30.165772 1541 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDHvgV0/build/release/bin/kudu as pid 24274
I20260923 22:21:30.165887 1541 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDHvgV0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201819212015-1541-0/minicluster-data/master-2/wal/instance
I20260923 22:21:30.166090 1541 external_mini_cluster.cc:1428] Setting key 0146b3d4af4005a3bce138e85419077d
I20260923 22:21:30.166704 24342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:21:30.177768 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.167706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.1.129.126:39479)
0923 22:21:30.167775 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.167777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.167925 (+ 148us) server_negotiation.cc:407] Connection header received
0923 22:21:30.167991 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.167992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.168000 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.168047 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.168539 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.169246 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.170464 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.170775 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.170864 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.173275 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.173276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.173277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.173284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.175554 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.176079 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.176080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.176081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.176151 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.176469 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.176470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.176471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.176732 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.176850 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.177556 (+ 706us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.177669 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:21:30.178313 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.167650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.126:39479 (local address 127.0.0.1:35526)
0923 22:21:30.167811 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.167853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.167928 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.168075 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.168078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.168365 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.168457 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.168462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.169332 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.169333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.170389 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.170392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.170881 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.170882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.170974 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.171634 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.171644 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.173186 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.175661 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.175667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.175669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.175993 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.176239 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.176240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.176241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.176383 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.176919 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.176925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.177269 (+ 344us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.178010 (+ 741us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.178160 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260923 22:21:30.179013 24342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:30.181509 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.166562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.1.129.124:35475)
0923 22:21:30.166752 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.166760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.166794 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:30.167211 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.167218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.167261 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.167437 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.168203 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.169449 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.170463 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.170950 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.171007 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.174926 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.174933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.174933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.174939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.180304 (+ 5365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.180668 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.180671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.180671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.180738 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.180940 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.180941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.180941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.181079 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.181221 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.181314 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.181388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.126@KRBTEST.COM
I20260923 22:21:30.187143 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.179503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.125:40045 (local address 127.0.0.1:38506)
0923 22:21:30.179534 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.179543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.179607 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.179790 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.179790 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.179954 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.179993 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.180008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.180713 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.180714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.181699 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.181701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.182221 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.182222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.182302 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.182949 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.182954 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.184287 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.186034 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.186034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.186035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.186255 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.186469 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.186471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.186471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.186535 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.186829 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.186830 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.186872 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.186970 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.187059 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260923 22:21:30.187148 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.179310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.1.129.125:40045)
0923 22:21:30.179368 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.179371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.179585 (+ 214us) server_negotiation.cc:407] Connection header received
0923 22:21:30.179706 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.179707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.179717 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.179785 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.180077 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.180612 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.181793 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.182109 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.182152 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.184365 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.184366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.184366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.184372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.185949 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.186338 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.186339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.186340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.186399 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.186609 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.186610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.186610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.186763 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.186822 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.186986 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.187078 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:21:30.187706 24342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:30.190013 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.182325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.1.129.126:39479)
0923 22:21:30.182367 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.182368 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.182446 (+ 78us) server_negotiation.cc:407] Connection header received
0923 22:21:30.182557 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.182558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.182567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.182659 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.183176 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.183743 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.184647 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.184911 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.184961 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.187515 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.187516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.187516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.187522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.188870 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.189280 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.189281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.189282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.189336 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.189536 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.189536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.189537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.189635 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.189787 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.189850 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.189926 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.124@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:21:30.194281 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.188020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.1.129.124:35475)
0923 22:21:30.188051 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.188053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.188061 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:30.188111 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.188111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.188118 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.188157 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.188424 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.188863 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.189775 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.190071 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.190136 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.191843 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.191844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.191844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.191850 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.193192 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.193612 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.193613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.193613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.193667 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.193878 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.193879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.193879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.193964 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.194011 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.194173 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.194232 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 22:21:30.194381 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.187969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36130)
0923 22:21:30.188008 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.188017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.188052 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.188184 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.188184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.188336 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.188365 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.188367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.188938 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.188939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.189684 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.189687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.190179 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.190180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.190226 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.190623 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.190627 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.191767 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.193289 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.193290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.193290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.193542 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.193747 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.193748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.193748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.193811 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.194027 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.194028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.194075 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.194155 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.194296 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":51}
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.129.125@KRBTEST.COM
I20260923 22:21:30.195829 24342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d: Bootstrap starting.
I20260923 22:21:30.196750 24342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:30.197023 24342 log.cc:824] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:30.197749 24342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d: No bootstrap required, opened a new log
I20260923 22:21:30.197986 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.190878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.1.129.125:40045)
0923 22:21:30.190943 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.190945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.190977 (+ 32us) server_negotiation.cc:407] Connection header received
0923 22:21:30.191044 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.191045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.191054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.191102 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.191602 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.192127 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.192757 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.192966 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.193050 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.195487 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.195489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.195489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.195495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.196882 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.197286 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.197287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.197288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.197342 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.197532 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.197533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.197533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.197639 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.197793 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.197828 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.197894 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 22:21:30.199358 1541 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:30.199729 24342 raft_consensus.cc:348] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } }
I20260923 22:21:30.199880 24342 raft_consensus.cc:729] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d274b4138dc445eb9f289003f744bb9d, State: Initialized, Role: FOLLOWER
I20260923 22:21:30.199962 24342 consensus_queue.cc:260] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } }
I20260923 22:21:30.200274 24342 sys_catalog.cc:564] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:30.200294 24348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.200417 24348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:30.200178 24166 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ffb334943d25412899a4ba25648352b8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35536
W20260923 22:21:30.202090 24359 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:30.202139 24359 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:30.202421 24274 master_runner.cc:186] Error getting master registration for 127.1.129.126:39479: OK, instance_id { permanent_uuid: "ffb334943d25412899a4ba25648352b8" instance_seqno: 1790202089869218 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:21:30.204674 24274 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:21:30.212476 24192 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } attempt: 2
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.126@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:21:30.220086 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.212738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36142)
0923 22:21:30.212774 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.212785 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.212861 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.213021 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.213022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.213201 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.213244 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.213245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.214148 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.214148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.214974 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.214976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.215438 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.215439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.215489 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.215943 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.215948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.217210 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.219007 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.219008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.219008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.219265 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.219469 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.219470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.219470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.219531 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.219786 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.219786 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.219825 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.219889 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.219989 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260923 22:21:30.220114 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.212785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.1.129.124:35475)
0923 22:21:30.212829 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.212832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.212845 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:21:30.212929 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.212930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.212940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.213009 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.213322 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.214062 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.215053 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.215359 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.215431 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.217288 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.217289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.217289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.217296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.218927 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.219339 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.219340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.219341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.219400 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.219603 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.219604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.219604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.219724 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.219786 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.219956 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.220027 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:21:30.221454 24192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8: Bootstrap starting.
I20260923 22:21:30.222066 24192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:30.222353 24192 log.cc:824] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:30.223081 24192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8: No bootstrap required, opened a new log
I20260923 22:21:30.224021 24192 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } }
I20260923 22:21:30.224148 24192 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb334943d25412899a4ba25648352b8, State: Initialized, Role: FOLLOWER
I20260923 22:21:30.224231 24192 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } }
I20260923 22:21:30.224479 24362 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.224537 24192 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:30.224551 24362 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:30.224663 24166 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 22:21:30.224743 24166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:30.225584 24166 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } }
I20260923 22:21:30.225827 24166 leader_election.cc:288] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [CANDIDATE]: Term 1 election: Requested vote from peers 524c2a74940d48f58613f53a50e41270 (127.1.129.125:40045), d274b4138dc445eb9f289003f744bb9d (127.1.129.124:35475)
I20260923 22:21:30.226828 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffb334943d25412899a4ba25648352b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d274b4138dc445eb9f289003f744bb9d"
I20260923 22:21:30.226951 24316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:30.227813 24316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb334943d25412899a4ba25648352b8 in term 1.
W20260923 22:21:30.228749 24373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:30.228812 24373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:30.227451 24124 master_runner.cc:186] Error getting master registration for 127.1.129.125:40045: OK, instance_id { permanent_uuid: "524c2a74940d48f58613f53a50e41270" instance_seqno: 1790202090013493 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:21:30.229957 24124 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:21:30.230167 24141 leader_election.cc:302] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d274b4138dc445eb9f289003f744bb9d, ffb334943d25412899a4ba25648352b8; no voters:
I20260923 22:21:30.230278 24362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:30.230417 24362 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [term 1 LEADER]: Becoming Leader. State: Replica: ffb334943d25412899a4ba25648352b8, State: Running, Role: LEADER
I20260923 22:21:30.230502 24362 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } }
I20260923 22:21:30.230932 24362 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffb334943d25412899a4ba25648352b8. Latest consensus state: current_term: 1 leader_uuid: "ffb334943d25412899a4ba25648352b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.231031 24362 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:30.232405 24384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:30.232587 24384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:30.237406 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.226941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.1.129.126:39479)
0923 22:21:30.227001 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.227003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.228963 (+ 1960us) server_negotiation.cc:407] Connection header received
0923 22:21:30.229059 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.229060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.229070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.229099 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.229849 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.230589 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.231697 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.231988 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.232025 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.234179 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.234181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.234181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.234187 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.235785 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.236182 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.236183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.236184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.236233 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.236411 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.236411 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.236412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.236521 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.236612 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.237246 (+ 634us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.237319 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:21:30.239301 24316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [term 1 FOLLOWER]: Refusing update from remote peer ffb334943d25412899a4ba25648352b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:30.239609 24382 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:30.240823 24348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffb334943d25412899a4ba25648352b8. Latest consensus state: current_term: 1 leader_uuid: "ffb334943d25412899a4ba25648352b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.240901 24348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:30.242321 24362 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffb334943d25412899a4ba25648352b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.242404 24362 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:30.242558 24385 mvcc.cc:205] Tried to move back new op lower bound from 7332667761602461696 to 7332667761584889856. Current Snapshot: MvccSnapshot[applied={T|T < 7332667761602461696}]
I20260923 22:21:30.242838 24348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffb334943d25412899a4ba25648352b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.242906 24348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:30.243386 24386 mvcc.cc:205] Tried to move back new op lower bound from 7332667761602461696 to 7332667761584889856. Current Snapshot: MvccSnapshot[applied={T|T < 7332667761602461696}]
I20260923 22:21:30.245728 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.228209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.1.129.125:40045)
0923 22:21:30.230102 (+ 1893us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.230104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.237092 (+ 6988us) server_negotiation.cc:407] Connection header received
0923 22:21:30.237115 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.237116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.237126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.237182 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.238325 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.239078 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.240066 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.240385 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.240457 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.242312 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.242313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.242314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.242320 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.244093 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.244503 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.244504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.244504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.244577 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.244932 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.244932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.244933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.245062 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.245147 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.245431 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.245562 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1868}
I20260923 22:21:30.247586 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.227379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.1.129.124:35475)
0923 22:21:30.227411 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.227413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.231348 (+ 3935us) server_negotiation.cc:407] Connection header received
0923 22:21:30.231485 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.231486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.231495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.231524 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.233032 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.233731 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.235786 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.236143 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.236183 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.244626 (+ 8443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.244627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.244628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.244634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.246240 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.246717 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.246718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.246718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.246771 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.246971 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.246972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.246972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.247123 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.247239 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.247426 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.247500 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:30.251103 24384 catalog_manager.cc:1347] Generated new cluster ID: 163c9a97fc974097a988bad39433d90c
I20260923 22:21:30.251158 24384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:30.251302 24267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } attempt: 3
Sep 23 22:21:30 dist-test-slave-dmw7 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.129.125@KRBTEST.COM for kudu/127.1.129.124@KRBTEST.COM
I20260923 22:21:30.267304 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.251523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.129.124:35475 (local address 127.0.0.1:36168)
0923 22:21:30.251612 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.251623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.251664 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.251830 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.251831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.251974 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.252009 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.252011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.252826 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.252826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.253591 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.253593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.254256 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.254257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.254310 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.254748 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.254753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.256141 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.258200 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.258201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.258201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.258454 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.258739 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.258740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.258740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.258809 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.266953 (+ 8144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.266954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.267001 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.267073 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.267185 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 22:21:30.267321 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.251573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.1.129.124:35475)
0923 22:21:30.251726 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.251728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.251734 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:21:30.251752 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.251752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.251760 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.251799 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.252073 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.252751 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.253887 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.254155 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.254213 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.256238 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.256239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.256239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.256246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.258104 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.258563 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.258565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.258566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.258636 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.266702 (+ 8066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.266703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.266704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.266849 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.266923 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.267126 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.267226 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260923 22:21:30.269452 24267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270: Bootstrap starting.
I20260923 22:21:30.270215 24267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:30.270516 24267 log.cc:824] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:30.277189 24267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270: No bootstrap required, opened a new log
I20260923 22:21:30.279184 24267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } }
I20260923 22:21:30.279320 24267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524c2a74940d48f58613f53a50e41270, State: Initialized, Role: FOLLOWER
I20260923 22:21:30.279399 24267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } }
I20260923 22:21:30.279722 24267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:30.279752 24241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffb334943d25412899a4ba25648352b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "524c2a74940d48f58613f53a50e41270"
I20260923 22:21:30.279861 24241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:30.280716 24241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb334943d25412899a4ba25648352b8 in term 1.
I20260923 22:21:30.280900 24240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [term 1 FOLLOWER]: Refusing update from remote peer ffb334943d25412899a4ba25648352b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:30.281103 24384 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:30.281337 24382 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:30.281821 24384 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:30.283104 24392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.283396 24392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:30.283533 24392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffb334943d25412899a4ba25648352b8. Latest consensus state: current_term: 1 leader_uuid: "ffb334943d25412899a4ba25648352b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.283674 24392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:30.283757 24392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffb334943d25412899a4ba25648352b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.283854 24392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:30.285960 24384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8: Generated new TSK 0
I20260923 22:21:30.286171 24384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:30.286414 24404 mvcc.cc:205] Tried to move back new op lower bound from 7332667761772544000 to 7332667761584889856. Current Snapshot: MvccSnapshot[applied={T|T < 7332667761602461696}]
W20260923 22:21:30.286968 24407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:30.287019 24407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:30.288012 24389 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffb334943d25412899a4ba25648352b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb334943d25412899a4ba25648352b8" member_type: VOTER last_known_addr { host: "127.1.129.126" port: 39479 } } peers { permanent_uuid: "524c2a74940d48f58613f53a50e41270" member_type: VOTER last_known_addr { host: "127.1.129.125" port: 40045 } } peers { permanent_uuid: "d274b4138dc445eb9f289003f744bb9d" member_type: VOTER last_known_addr { host: "127.1.129.124" port: 35475 } } }
I20260923 22:21:30.288101 24389 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb334943d25412899a4ba25648352b8 [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:30.313186 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.311741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.1.129.125:40045)
0923 22:21:30.311786 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.311789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.312537 (+ 748us) server_negotiation.cc:407] Connection header received
0923 22:21:30.312643 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.312644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.312654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.312697 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.313119 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38526: BlockingRecv error: recv got EOF from 127.0.0.1:38526 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 22:21:30.313886 24195 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.312107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.1.129.126:39479)
0923 22:21:30.312144 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.312146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.313411 (+ 1265us) server_negotiation.cc:407] Connection header received
0923 22:21:30.313502 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.313503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.313514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.313542 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.313830 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35544: BlockingRecv error: recv got EOF from 127.0.0.1:35544 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 22:21:30.314543 24345 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.312395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.1.129.124:35475)
0923 22:21:30.312436 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.312438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.314086 (+ 1648us) server_negotiation.cc:407] Connection header received
0923 22:21:30.314178 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.314179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.314189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.314218 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.314489 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36178: BlockingRecv error: recv got EOF from 127.0.0.1:36178 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 22:21:30.316769 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.316096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.1.129.125:40045)
0923 22:21:30.316132 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.316134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.316291 (+ 157us) server_negotiation.cc:407] Connection header received
0923 22:21:30.316389 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.316390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.316406 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.316434 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.316730 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38530: BlockingRecv error: recv got EOF from 127.0.0.1:38530 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:21:30.319564 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.318906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.1.129.125:40045)
0923 22:21:30.318934 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.318935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.319144 (+ 209us) server_negotiation.cc:407] Connection header received
0923 22:21:30.319229 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.319230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.319239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.319265 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.319523 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38532: BlockingRecv error: recv got EOF from 127.0.0.1:38532 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260923 22:21:30.323290 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.322502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.1.129.125:40045)
0923 22:21:30.322532 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.322533 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.322828 (+ 295us) server_negotiation.cc:407] Connection header received
0923 22:21:30.322920 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.322921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.322931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.322958 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.323253 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38546: BlockingRecv error: recv got EOF from 127.0.0.1:38546 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260923 22:21:30.329053 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.328291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.1.129.125:40045)
0923 22:21:30.328324 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.328325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.328558 (+ 233us) server_negotiation.cc:407] Connection header received
0923 22:21:30.328654 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.328655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.328678 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.328709 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.329013 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38560: BlockingRecv error: recv got EOF from 127.0.0.1:38560 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:21:30.339203 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.338205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.1.129.125:40045)
0923 22:21:30.338244 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.338251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.338525 (+ 274us) server_negotiation.cc:407] Connection header received
0923 22:21:30.338723 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.338724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.338736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.338770 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.339139 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38568: BlockingRecv error: recv got EOF from 127.0.0.1:38568 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 22:21:30.357424 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.356483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.1.129.125:40045)
0923 22:21:30.356531 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.356533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.356829 (+ 296us) server_negotiation.cc:407] Connection header received
0923 22:21:30.356938 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.356939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.356950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.356981 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.357364 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38576: BlockingRecv error: recv got EOF from 127.0.0.1:38576 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 22:21:30.391644 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.390741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.1.129.125:40045)
0923 22:21:30.390784 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.390785 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.391064 (+ 279us) server_negotiation.cc:407] Connection header received
0923 22:21:30.391162 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.391163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.391176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.391211 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.391587 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38592: BlockingRecv error: recv got EOF from 127.0.0.1:38592 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 22:21:30.457767 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.457043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.1.129.125:40045)
0923 22:21:30.457110 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.457112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.457224 (+ 112us) server_negotiation.cc:407] Connection header received
0923 22:21:30.457254 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.457255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.457264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.457329 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.457671 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38598: BlockingRecv error: recv got EOF from 127.0.0.1:38598 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 22:21:30.587673 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.586967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.1.129.125:40045)
0923 22:21:30.587043 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.587045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.587095 (+ 50us) server_negotiation.cc:407] Connection header received
0923 22:21:30.587187 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.587188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.587198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.587250 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.587584 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38610: BlockingRecv error: recv got EOF from 127.0.0.1:38610 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260923 22:21:30.845501 24272 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:30.844835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.1.129.125:40045)
0923 22:21:30.844888 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.844890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.844931 (+ 41us) server_negotiation.cc:407] Connection header received
0923 22:21:30.845006 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.845007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.845017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.845066 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.845455 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38620: BlockingRecv error: recv got EOF from 127.0.0.1:38620 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:31.202675 24359 catalog_manager.cc:1259] Loaded cluster ID: 163c9a97fc974097a988bad39433d90c
I20260923 22:21:31.202767 24359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d: loading cluster ID for follower catalog manager: success
I20260923 22:21:31.203543 24359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d: acquiring CA information for follower catalog manager: success
I20260923 22:21:31.203792 24359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d274b4138dc445eb9f289003f744bb9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:21:31.287488 24407 catalog_manager.cc:1259] Loaded cluster ID: 163c9a97fc974097a988bad39433d90c
I20260923 22:21:31.287571 24407 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270: loading cluster ID for follower catalog manager: success
I20260923 22:21:31.288363 24407 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270: acquiring CA information for follower catalog manager: success
I20260923 22:21:31.288635 24407 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 524c2a74940d48f58613f53a50e41270: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:21:31.361642 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.358737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.1.129.125:40045)
0923 22:21:31.358891 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.358893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.358907 (+ 14us) server_negotiation.cc:407] Connection header received
0923 22:21:31.358983 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.358983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.359033 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.359107 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:31.359313 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.359802 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.360642 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.360853 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.360925 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.361115 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:31.361231 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:31.361301 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.361406 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.361431 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260923 22:21:31.366138 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.363764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.1.129.124:35475)
0923 22:21:31.363857 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.363859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.363889 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:21:31.363956 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.363957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.363991 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.364063 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:31.364166 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.364649 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.365382 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.365606 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.365676 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.365812 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:31.365924 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:31.365989 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.366025 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.366046 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":52,"threads_started":1}
I20260923 22:21:31.370738 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.368185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.1.129.124:35475)
0923 22:21:31.368223 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.368225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.368259 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:31.368327 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.368327 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.368338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.368377 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:31.368506 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.369209 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.369969 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.370214 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.370302 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.370459 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:31.370490 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:31.370537 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.370649 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.370670 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:31.371281 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.368231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.1.129.126:39479)
0923 22:21:31.368455 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.368457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.368465 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:31.368510 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.368511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.368559 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.368661 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:31.368767 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.369273 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.370081 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.370397 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.370492 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.370680 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:31.370783 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:31.371138 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.371156 (+ 18us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.371180 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":66,"threads_started":1}
I20260923 22:21:31.371879 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.367959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.1.129.125:40045)
0923 22:21:31.368013 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.368016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.368124 (+ 108us) server_negotiation.cc:407] Connection header received
0923 22:21:31.368230 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.368231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.368244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.368275 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:31.368466 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.369209 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.370901 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.371189 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.371230 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.371423 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:31.371456 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:31.371502 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.371743 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.371774 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:21:31.376618 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 24124
I20260923 22:21:31.382226 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 24199
I20260923 22:21:31.387132 1541 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDHvgV0/build/release/bin/kudu with pid 24274
2026-09-23T22:21:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2789 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5595 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (272182 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST